Senior Software Engineer, Data Engineering
ValGenesis10 months ago
Chennai, IndiaMid Level / Senior
Responsibilities
- Design, develop, and maintain batch and real-time data ingestion, transformation, and orchestration pipelines.
- Build and optimize lakehouse architectures using Azure Synapse, Delta Lake, or similar frameworks.
- Integrate and manage structured and unstructured data from SQL/NoSQL systems, files, documents, and IoT streams.
- Develop and operationalize ETL/ELT pipelines using Azure Data Factory, Databricks, or Apache Spark.
- Prepare and serve ML-ready datasets for model training and inference in collaboration with data scientists.
- Implement data quality, lineage, governance, monitoring, observability, scalability, and automation across data workflows.
- Enable self-service analytics with Power BI, Superset, and Tableau.
- Deploy and maintain data APIs and ML models in production using Azure ML, Kubernetes, and CI/CD pipelines.
- Collaborate with engineering, product, and business teams to translate insights into action.
Requirements
- 4 to 8 years of experience in Data Engineering or equivalent roles.
- Strong Python and SQL skills plus at least one compiled language: C#, Java, or Scala.
- Experience with relational databases including SQL Server, PostgreSQL, and MySQL.
- Experience with NoSQL systems including MongoDB and Cosmos DB.
- Hands-on experience with Azure Data Lake and Databricks.
- Experience with Azure Data Factory, Apache Airflow, or dbt for ETL/ELT.
- Experience with Kafka, Event Hubs, or Service Bus for real-time processing.
- Familiarity with TensorFlow, PyTorch, and MLOps concepts.
- Experience with Power BI, Apache Superset, or Tableau.
- Experience with Azure, Docker, Kubernetes, and GitHub Actions or Azure DevOps.
- Understanding of data modeling, version control, and CI/CD for data systems.
- Preferred experience with knowledge graphs, semantic search, LLM-based RAG patterns, data mesh, data fabric, domain-oriented data architecture, MLflow, Delta Live Tables, or Databricks Unity Catalog.
- Strong analytical, problem-solving, communication, collaboration, and attention-to-detail skills.
Benefits
- Chennai, Hyderabad, and Bangalore office roles are onsite five days per week.
- Equal-opportunity employment with consideration based on merit and protected characteristics.
Tech Stack
Apache AirflowApache KafkaApache SparkApache SupersetAzureC#DatabricksdbtDockerGitHub ActionsJavaKubernetesMicrosoft SQL ServerMLflowMongoDBMySQLPostgreSQLPythonPyTorchScalaSQLTensorFlow
Categories
Data Engineering