Lead Software Engineer, Data Engineering
ValGenesis11 months ago
Chennai, IndiaSenior / Staff+
Responsibilities
- Architect, design, develop, and maintain batch and real-time data ingestion, transformation, and orchestration pipelines.
- Build and optimize lakehouse architectures using Azure Synapse, Delta Lake, or similar frameworks.
- Integrate structured and unstructured data from databases, files, documents, and IoT streams.
- Develop and operationalize ETL/ELT pipelines using Azure Data Factory, Databricks, or Apache Spark.
- Prepare and serve ML-ready datasets for model training and inference in collaboration with data scientists.
- Implement data quality, lineage, governance, monitoring, automation, scalability, and observability across data workflows.
- Enable self-service analytics through BI tools and deploy production data APIs and ML models.
- Collaborate with engineering, product, and business teams and mentor junior team members.
Requirements
- 6 to 12 years of experience in Data Engineering or equivalent roles.
- Strong Python and SQL skills plus at least one compiled language: C#, Java, or Scala.
- Experience with relational databases including SQL Server, PostgreSQL, and MySQL, and NoSQL systems including MongoDB and Cosmos DB.
- Hands-on experience with Azure Data Lake and Databricks.
- Experience with Azure Data Factory, Apache Airflow, or dbt for ETL/ELT.
- Experience with Kafka, Event Hubs, or Service Bus for real-time data processing.
- Familiarity with TensorFlow, PyTorch, and MLOps concepts.
- Experience with Power BI, Apache Superset, or Tableau.
- Experience with Azure, Docker, Kubernetes, and GitHub Actions or Azure DevOps.
- Understanding of data modeling, version control, and CI/CD for data systems.
- Preferred experience with knowledge graphs, semantic search, LLM-based data retrieval and RAG patterns, data mesh, data fabric, MLflow, Delta Live Tables, or Databricks Unity Catalog.
- Strong analytical, problem-solving, communication, collaboration, and attention-to-detail skills.
Benefits
- The Chennai, Hyderabad, and Bangalore offices require onsite work five days per week.
- ValGenesis is an equal-opportunity employer and considers qualified applicants without regard to legally protected characteristics.
Tech Stack
Apache AirflowApache KafkaApache SparkApache SupersetAzureC#DatabricksdbtDockerGitHub ActionsJavaKubernetesMicrosoft SQL ServerMLflowMongoDBMySQLPostgreSQLPythonPyTorchScalaSQLTensorFlow
Categories
Data Engineering
About ValGenesis
ValGenesis builds enterprise software for digitizing validation and manufacturing workflows in the life sciences, centered on its Validation Lifecycle Management System (VLMS) and eLog tools. Customers use it to manage equipment, cleaning, process, and computer system validation with audit trails and compliance support. Founded in 2005 and headquartered in Santa Clara, the privately held firm says its products are used by 30 of the top 50 global pharma and biotech companies.