Senior Software Engineer, Data Engineering
ValGenesis6 months ago
Chennai, IndiaMid Level / Senior
Responsibilities
- Design, develop, and maintain batch and real-time data ingestion, transformation, and orchestration pipelines.
- Build and optimize Lakehouse architectures using Azure Synapse, Delta Lake, or similar frameworks.
- Integrate and manage structured, unstructured, document, file, SQL/NoSQL, and IoT stream data sources.
- Develop and operationalize ETL/ELT pipelines with Azure Data Factory, Databricks, or Apache Spark.
- Prepare and serve ML-ready datasets for model training and inference in collaboration with data scientists.
- Implement data quality, lineage, governance, monitoring, automation, scalability, performance, and observability for data workflows.
- Enable self-service analytics through Power BI, Apache Superset, and Tableau.
- Deploy and maintain data APIs and ML models in production using Azure ML, Kubernetes, and CI/CD pipelines.
- Collaborate with engineering, product, business, and cross-functional teams to translate insights into action.
Requirements
- 4 to 8 years of experience in data engineering or equivalent roles.
- Strong programming ability in Python, SQL, and at least one compiled language: C#, Java, or Scala.
- Experience with relational databases including SQL Server, PostgreSQL, and MySQL.
- Experience with NoSQL systems including MongoDB and Cosmos DB.
- Hands-on experience with Azure Data Lake and Databricks.
- Experience with Azure Data Factory, Apache Airflow, or dbt for ETL/ELT.
- Experience with Kafka, Event Hubs, or Service Bus for real-time data processing.
- Familiarity with TensorFlow, PyTorch, and MLOps concepts.
- Experience with Power BI, Apache Superset, or Tableau.
- Experience with Azure, Docker, Kubernetes, and GitHub Actions or Azure DevOps.
- Understanding of data modeling, version control, and CI/CD for data systems.
- Preferred experience with knowledge graphs, semantic search, LLM-based RAG patterns, data mesh, data fabric, domain-oriented data architecture, MLflow, Delta Live Tables, or Databricks Unity Catalog.
Benefits
- Chennai, Hyderabad, and Bangalore office roles are onsite five days per week.
- ValGenesis is an equal-opportunity employer.
Tech Stack
Apache AirflowApache KafkaApache SparkApache SupersetAzureC#DatabricksdbtDockerGitHub ActionsJavaKubernetesMicrosoft SQL ServerMLflowMongoDBMySQLPostgreSQLPythonPyTorchScalaSQLTensorFlow
Categories
Data Engineering
About ValGenesis
ValGenesis builds enterprise software for digitizing validation and manufacturing workflows in the life sciences, centered on its Validation Lifecycle Management System (VLMS) and eLog tools. Customers use it to manage equipment, cleaning, process, and computer system validation with audit trails and compliance support. Founded in 2005 and headquartered in Santa Clara, the privately held firm says its products are used by 30 of the top 50 global pharma and biotech companies.