Sr. Data Platform Engineer
Thermo Fisher Scientific1 month ago
Bengaluru, IndiaSenior
Responsibilities
- Configure and manage Databricks workspaces, clusters, policies, runtimes, and separate development, QA/UAT, and production environments.
- Implement Databricks security and governance using Unity Catalog, IAM/RBAC, service principals, secrets, private connectivity, and enterprise access controls.
- Build and maintain scalable ETL/ELT pipelines and Lakehouse solutions using Apache Spark, PySpark, Databricks workflows/jobs, and Delta Lake.
- Automate Databricks infrastructure and deployments with Terraform, source control, and CI/CD practices.
- Optimize Spark workloads, queries, clusters, and resource usage for performance, reliability, scalability, and cost.
- Develop machine learning and generative AI workflows covering feature engineering, model training, evaluation, deployment, lifecycle management, LLMs, embeddings, vector search, RAG, prompt engineering, and model serving.
- Create analytical datasets, semantic models, KPIs, dashboards, and reporting layers using Databricks SQL and BI tools.
- Establish monitoring, logging, alerting, troubleshooting, and operational support for data, AI, generative AI, and analytics workloads.
- Collaborate with architecture, security, infrastructure, engineering, data science, analytics, DevOps, and business teams to define platform standards and best practices.
- Maintain technical documentation for platform configuration, deployments, pipelines, AI workflows, dashboards, and operational procedures.
Requirements
- At least 10 years of professional experience building analytics platforms, including at least 5 years of hands-on Databricks experience.
- A Databricks certification is required.
- Strong production experience with Databricks, Apache Spark/PySpark, Python, SQL, Delta Lake, Lakehouse architecture, workspaces, clusters, jobs/workflows, and scalable ETL/ELT pipelines.
- Experience with Azure, AWS, or Google Cloud, including cloud storage, Unity Catalog, IAM/RBAC, secrets management, networking, and secure connectivity.
- Experience with Terraform or similar Infrastructure-as-Code tools, CI/CD, and source-control practices.
- Working experience with MLflow and AI/ML/GenAI delivery, including model lifecycle, LLMs, embeddings, vector search, RAG, prompt engineering, and model serving.
- Experience with Databricks SQL and BI tools such as Power BI, Tableau, or Looker, including semantic models and KPIs.
- Strong troubleshooting, performance optimization, monitoring, and production-support skills.
- Preferred experience includes enterprise-scale Databricks Lakehouse platforms, standardized multi-environment deployments, Databricks Asset Bundles, Apache Airflow, Azure Data Factory, Spark Structured Streaming, Kafka, Event Hubs, LangChain, LlamaIndex, Hugging Face, Azure OpenAI, Amazon Bedrock, fine-tuning, model evaluation, data governance, metadata management, data-quality frameworks, BI governance, privacy controls, responsible AI, and regulated or large enterprise environments.
- Bachelor’s or master’s degree in computer science, information technology, engineering, or a related discipline.
Benefits
- Standard Monday-Friday work schedule.
- Office-based work environment.
Tech Stack
Categories
Data EngineeringML Engineering
About Thermo Fisher Scientific
Thermo Fisher Scientific builds analytical instruments, lab equipment and consumables, specialty diagnostics, and provides pharma services (CDMO and clinical research) for biotech, pharma, academic, and healthcare customers. It earns revenue through product sales, e-commerce distribution (Fisher Scientific), and contract services under brands such as Thermo Scientific, Applied Biosystems, Invitrogen, Patheon, and PPD. Headquartered in Waltham, MA, it is publicly traded on the NYSE (TMO) and was created by the 2006 merger of Thermo Electron and Fisher Scientific.