Data Engineer
Particle412 months ago
Remote, IndiaMid Level
Responsibilities
- Design, develop, and maintain scalable ETL pipelines processing large volumes of data from diverse sources.
- Build and optimize data lakes and data warehouses for efficient data retrieval and processing.
- Integrate structured and unstructured data from internal and external systems.
- Perform data validation, cleansing, transformation, and quality checks.
- Gather requirements and translate business needs into technical data solutions.
- Participate in Agile planning, stand-ups, reviews, and software delivery activities.
- Write unit and integration tests, debug applications, and resolve defects and performance bottlenecks.
- Document data processes, tools, and systems while promoting engineering best practices.
- Collaborate with product managers, testers, and other cross-functional teams.
- Evaluate new technologies and optimize existing code and data systems for performance, security, scalability, and maintainability.
Requirements
- Bachelor's degree in computer science, engineering, or a related field.
- At least 3 years of experience as a Data Engineer.
- Proficiency in Python.
- Experience with SQL databases such as MySQL and PostgreSQL and NoSQL databases such as MongoDB.
- Understanding of Flask, API frameworks, database and ORM technologies, data warehousing and lakehouse principles, Databricks, pandas, Spark, PySpark, machine learning, OpenCV, and scikit-learn.
- Experience with utilities and tools including logging, requests, subprocess, regex, pytest, the ELK stack, Redis, and distributed task queues.
- Understanding of concurrent and parallel processing concepts.
- Familiarity with at least one cloud data engineering stack among Azure, AWS, or Google Cloud Platform, plus the ability to learn ETL/ELT tools across cloud providers.
- Familiarity with Git and collaborative development workflows.
- Competence with Linux and shell scripting.
- Understanding of software engineering principles, design patterns, and best practices.
- Strong problem-solving, analytical, communication, collaboration, adaptability, and attention-to-detail skills.
Tech Stack
Apache SparkAWSAzureDatabricksFlaskGitGoogle Cloud PlatformLinuxMongoDBMySQLOpenCVPandasPostgreSQLpytestPythonRedisscikit-learnSQL
Categories
Data Engineering
About Particle41
Particle41 is a software engineering consultancy that embeds product, DevOps, and data teams to build and modernize applications for startups and enterprises. It offers application development, cloud-native DevOps, data engineering/data science, and fractional CTO advisory on a project or retainer basis. Founded in 2015 and headquartered in Frisco, Texas, the privately held firm works alongside client engineering leadership to deliver in-house solutions.