Data Engineer (Elasticsearch + Datawarehousing)
Particle41over 1 year ago
Remote, IndiaMid Level
Responsibilities
- Design, implement, and optimize Elasticsearch clusters, indexes, and data retrieval solutions.
- Build and optimize data lakes, data warehouses, and integrated structured and unstructured data solutions.
- Perform data validation, cleansing, transformation, and quality checks.
- Develop APIs, microservices, backend systems, automation scripts, and integrations with Elasticsearch and databases.
- Gather requirements and translate stakeholder and business needs into technical solutions.
- Conduct unit and integration testing, debug software, and resolve defects, code smells, and performance bottlenecks.
- Maintain documentation and promote best practices for data processes, tools, and systems.
- Participate in Agile planning, stand-ups, reviews, and collaborative delivery with cross-functional teams.
- Stay current with cloud platforms and propose improvements to application performance, security, scalability, and maintainability.
Requirements
- Bachelor's degree in Computer Science, Engineering, or a related field.
- At least 3 years of experience as a Data Engineer.
- Proficiency in Elasticsearch and Python programming.
- Experience with SQL databases such as MySQL and PostgreSQL, and NoSQL databases such as MongoDB.
- Understanding of Flask, data warehousing and lakehousing, Databricks, Pandas, Spark, PySpark, machine learning, OpenCV, and Scikit-learn.
- Experience using Java for backend systems and Elasticsearch or database integrations.
- Familiarity with the ELK stack, Redis, distributed task queues, concurrent and parallel processing, and cloud data engineering stacks on Azure, AWS, or GCP.
- Familiarity with Git, Linux, shell scripting, and collaborative development workflows.
- Understanding of software engineering principles, design patterns, APIs, databases, ORM, and best practices.
- Strong problem-solving, analytical, communication, collaboration, adaptability, and attention-to-detail skills.
Tech Stack
Apache SparkAWSAzureDatabricksElasticsearchFlaskGitGoogle Cloud PlatformJavaLinuxMongoDBMySQLOpenCVPandasPostgreSQLpytestPythonRedisscikit-learnSQL
Categories
BackendData Engineering
About Particle41
Particle41 is a software engineering consultancy that embeds product, DevOps, and data teams to build and modernize applications for startups and enterprises. It offers application development, cloud-native DevOps, data engineering/data science, and fractional CTO advisory on a project or retainer basis. Founded in 2015 and headquartered in Frisco, Texas, the privately held firm works alongside client engineering leadership to deliver in-house solutions.