Senior Software Engineer, Data Engineering
ValGenesis4 months ago
Chennai, IndiaMid Level / Senior
Responsibilities
- Design, develop, and maintain batch and real-time data ingestion, transformation, and orchestration pipelines.
- Build and optimize Lakehouse architectures using Azure Synapse, Delta Lake, or similar frameworks.
- Integrate and manage structured, unstructured, document, file, SQL/NoSQL, and IoT stream data sources.
- Develop and operationalize ETL/ELT pipelines with Azure Data Factory, Databricks, or Apache Spark.
- Prepare and serve ML-ready datasets for model training and inference in collaboration with data scientists.
- Implement data quality, lineage, governance, monitoring, automation, scalability, performance, and observability for data workflows.
- Enable self-service analytics through Power BI, Apache Superset, and Tableau.
- Deploy and maintain data APIs and ML models in production using Azure ML, Kubernetes, and CI/CD pipelines.
- Collaborate with engineering, product, business, and cross-functional teams to translate insights into action.
Requirements
- 4 to 8 years of experience in data engineering or equivalent roles.
- Strong programming ability in Python, SQL, and at least one compiled language: C#, Java, or Scala.
- Experience with relational databases including SQL Server, PostgreSQL, and MySQL.
- Experience with NoSQL systems including MongoDB and Cosmos DB.
- Hands-on experience with Azure Data Lake and Databricks.
- Experience with Azure Data Factory, Apache Airflow, or dbt for ETL/ELT.
- Experience with Kafka, Event Hubs, or Service Bus for real-time data processing.
- Familiarity with TensorFlow, PyTorch, and MLOps concepts.
- Experience with Power BI, Apache Superset, or Tableau.
- Experience with Azure, Docker, Kubernetes, and GitHub Actions or Azure DevOps.
- Understanding of data modeling, version control, and CI/CD for data systems.
- Preferred experience with knowledge graphs, semantic search, LLM-based RAG patterns, data mesh, data fabric, domain-oriented data architecture, MLflow, Delta Live Tables, or Databricks Unity Catalog.
Benefits
- Chennai, Hyderabad, and Bangalore office roles are onsite five days per week.
- ValGenesis is an equal-opportunity employer.
Tech Stack
Apache AirflowApache KafkaApache SparkApache SupersetAzureC#DatabricksdbtDockerGitHub ActionsJavaKubernetesMicrosoft SQL ServerMLflowMongoDBMySQLPostgreSQLPythonPyTorchScalaSQLTensorFlow
Categories
Data Engineering