Lead Software Engineer, Data Engineering
ValGenesis10 months ago
Chennai, IndiaSenior / Staff+
Responsibilities
- Architect, design, develop, and maintain batch and real-time data ingestion, transformation, and orchestration pipelines.
- Build and optimize lakehouse architectures using Azure Synapse, Delta Lake, or similar frameworks.
- Integrate structured and unstructured data from databases, files, documents, and IoT streams.
- Develop and operationalize ETL/ELT pipelines using Azure Data Factory, Databricks, or Apache Spark.
- Prepare and serve ML-ready datasets for model training and inference in collaboration with data scientists.
- Implement data quality, lineage, governance, monitoring, automation, scalability, and observability across data workflows.
- Enable self-service analytics through BI tools and deploy production data APIs and ML models.
- Collaborate with engineering, product, and business teams and mentor junior team members.
Requirements
- 6 to 12 years of experience in Data Engineering or equivalent roles.
- Strong Python and SQL skills plus at least one compiled language: C#, Java, or Scala.
- Experience with relational databases including SQL Server, PostgreSQL, and MySQL, and NoSQL systems including MongoDB and Cosmos DB.
- Hands-on experience with Azure Data Lake and Databricks.
- Experience with Azure Data Factory, Apache Airflow, or dbt for ETL/ELT.
- Experience with Kafka, Event Hubs, or Service Bus for real-time data processing.
- Familiarity with TensorFlow, PyTorch, and MLOps concepts.
- Experience with Power BI, Apache Superset, or Tableau.
- Experience with Azure, Docker, Kubernetes, and GitHub Actions or Azure DevOps.
- Understanding of data modeling, version control, and CI/CD for data systems.
- Preferred experience with knowledge graphs, semantic search, LLM-based data retrieval and RAG patterns, data mesh, data fabric, MLflow, Delta Live Tables, or Databricks Unity Catalog.
- Strong analytical, problem-solving, communication, collaboration, and attention-to-detail skills.
Benefits
- The Chennai, Hyderabad, and Bangalore offices require onsite work five days per week.
- ValGenesis is an equal-opportunity employer and considers qualified applicants without regard to legally protected characteristics.
Tech Stack
Apache AirflowApache KafkaApache SparkApache SupersetAzureC#DatabricksdbtDockerGitHub ActionsJavaKubernetesMicrosoft SQL ServerMLflowMongoDBMySQLPostgreSQLPythonPyTorchScalaSQLTensorFlow
Categories
Data Engineering