2 months ago
Hyderābād, IndiaSenior
Responsibilities
- Lead the Databricks platform roadmap, reference architecture, operating model, standards, guardrails, lifecycle strategy, and capability backlog.
- Evaluate Databricks features and third-party technologies through architecture assessments and proofs of concept.
- Design enterprise account, workspace, metastore, catalog, networking, storage, compute, deployment, high-availability, disaster-recovery, and capacity strategies.
- Build reusable platform frameworks, APIs, templates, provisioning workflows, policy-as-code controls, and reference implementations.
- Integrate Databricks with Collibra and other enterprise metadata, data-quality, observability, and governance services.
- Define security controls for identity, access, encryption, secrets, private connectivity, auditing, and data-exfiltration prevention.
- Lead Databricks FinOps, including tagging, budgets, chargeback or showback, compute optimization, usage policies, and cost-anomaly detection.
- Establish platform observability using SLOs, KPIs, telemetry, dashboards, alerts, system tables, audit logs, billing data, lineage, and MLflow tracing.
- Enable governed ML, GenAI, RAG, agent, Model Serving, AI Search, evaluation, monitoring, and LLMOps capabilities.
- Improve reliability through root-cause analysis, operational reviews, automated remediation, runbooks, resilience testing, and incident reduction.
- Drive onboarding, documentation, workshops, office hours, reference solutions, technical guidance, and platform-community adoption.
Requirements
- Bachelor's or master's degree in computer science or an engineering field.
- 9–12 years of relevant experience.
- Experience owning or technically leading an enterprise Databricks platform beyond notebook or pipeline development, including architecture, administration, governance, security, lifecycle management, reliability, and enablement.
- Expertise across Databricks administration, Apache Spark, Delta Lake, Unity Catalog, Lakeflow Jobs and Pipelines, Databricks SQL, Photon, AI/BI, MLflow, and Model Serving.
- Experience designing enterprise workspace, metastore, catalog, workload-isolation, promotion, high-availability, disaster-recovery, and capacity strategies.
- Strong Unity Catalog, identity, access-control, security, auditing, encryption, private-connectivity, and data-exfiltration-control expertise.
- Strong AWS experience with IAM, VPC, PrivateLink, S3, EC2, KMS, CloudWatch, CloudTrail, Secrets Manager, and STS; EKS, Lambda, Glue, EMR, and RDS are beneficial.
- Experience with FinOps, compute rightsizing, SQL warehouse optimization, budgets, tagging, billing data, system tables, telemetry, dashboards, alerts, and cost-anomaly investigation.
- Understanding of Databricks AI/ML foundations, feature engineering, inference, model monitoring, MLOps, RAG, agent evaluation, AI security, and LLM cost and performance.
- Strong Python, PySpark, SQL, automation, platform API, Databricks SDK, REST API, CLI, Terraform, Git, automated testing, promotion, rollback, and secrets-management skills.
- Strong Spark and SQL troubleshooting and performance-analysis skills, plus software engineering and distributed-systems fundamentals.
- Experience with Agile environments and tools such as Jira or Jira Align.
- Preferred experience includes Databricks agents, RAG, AI Gateway, Foundation Model APIs, Databricks AI/BI, Genie, Collibra, Ataccama, Monte Carlo, Datadog, Splunk, React, Docker, Kubernetes, Azure or GCP Databricks, and regulated-industry platforms.
- Preferred certifications include Databricks, AWS, and SAFe certifications.
- Excellent communication, influencing, coaching, documentation, planning, prioritization, systems-thinking, and consensus-building skills.
Benefits
- Competitive and comprehensive total rewards and benefits aligned with local industry standards.
- Full-time position with a collaborative, innovative, science-based culture and support for professional and personal growth.
Tech Stack
Apache SparkAWSAzureDatabricksDatadogDockerGitGoogle Cloud PlatformKubernetesMLflowPostgreSQLPythonReactSplunkSQLTerraform
Categories
Data EngineeringDevOps
About Amgen
Amgen is a public biotechnology company that discovers, develops, manufactures, and sells biologic and biosimilar therapeutics for patients with serious diseases. Its portfolio and pipeline span oncology, cardiovascular disease, osteoporosis, inflammation, and rare diseases, marketed globally through direct sales and partnerships. Founded in 1980 and headquartered in Thousand Oaks, California (NASDAQ: AMGN), Amgen is a component of the Dow Jones Industrial Average and completed the acquisition of Horizon Therapeutics in 2023.