1 month ago
Hyderābād, IndiaSenior
Responsibilities
- Lead the design and development of scalable batch and real-time ETL/ELT pipelines.
- Own complex data solutions from requirements and design through deployment and production support.
- Build cloud-based lakehouse and data-platform solutions using Databricks and AWS.
- Develop reusable, metadata-driven data integration frameworks.
- Integrate structured, semi-structured, and unstructured data from enterprise, manufacturing, API, and third-party sources.
- Optimize Spark workloads, Databricks compute, SQL queries, partitioning, storage, and caching for performance and cost.
- Implement workflow orchestration, monitoring, alerting, data-quality controls, and recovery processes.
- Develop CI/CD pipelines and automated testing for data solutions.
- Implement metadata management, lineage, cataloging, governance, RBAC, and data-security controls.
- Define data models, data contracts, integration patterns, and reusable engineering standards.
- Lead architecture reviews, code reviews, troubleshooting, and root-cause analysis.
- Mentor engineers and provide technical guidance across delivery teams.
- Collaborate with architects, analysts, data scientists, product teams, and DevOps teams.
- Support estimation, sprint planning, technical roadmaps, and delivery-risk management.
- Evaluate emerging technologies and recommend scalable, secure, maintainable solutions.
- Participate in operational support, including occasional off-hours support.
Requirements
- Master’s degree in Computer Science, Engineering, Information Technology, Data Science, or a related field and at least 7 years of relevant experience, or a bachelor’s degree in a related field and at least 9 years of relevant experience.
- Advanced hands-on experience with Databricks, Apache Spark, PySpark, Spark SQL, Delta Lake, Python, and SQL.
- Experience designing and operating production-grade batch and streaming pipelines.
- Strong understanding of distributed computing, lakehouse architecture, data warehousing, and data integration.
- Experience with Databricks Workflows or comparable orchestration tools and AWS data, compute, storage, security, and monitoring services.
- Experience with Spark performance tuning, cluster optimization, partitioning, cost management, Git, CI/CD, automated testing, monitoring, and production deployment.
- Experience implementing data quality, metadata management, lineage, governance, access controls, RBAC, least privilege, encryption, auditability, and regulated-data requirements.
- Experience leading technical design, code reviews, and complex production implementations.
- Ability to define reusable patterns, engineering standards, and development best practices.
- Strong communication, collaboration, mentoring, analytical, troubleshooting, and problem-solving skills.
- Experience working in Agile or Scaled Agile delivery environments.
- Preferred experience in biotechnology, pharmaceutical, life sciences, manufacturing, or another regulated industry.
- Preferred experience with Unity Catalog, data products, Data Fabric, Data Mesh, APIs, secure data services, relational databases, NoSQL databases, analytical databases, operational databases, vector databases, OLAP, OLTP, Kafka, Kinesis, RAG, embeddings, vector search, and governed enterprise data access.
- Familiarity with GitHub Copilot, OpenAI Codex, or equivalent AI-assisted development tools.
- Preferred AWS Certified Data Engineer or other relevant AWS certification, Databricks Certified Data Engineer Associate or Professional certification, or Scaled Agile Framework certification.
Tech Stack
Categories
Data Engineering
About Amgen
Amgen is a public biotechnology company that discovers, develops, manufactures, and sells biologic and biosimilar therapeutics for patients with serious diseases. Its portfolio and pipeline span oncology, cardiovascular disease, osteoporosis, inflammation, and rare diseases, marketed globally through direct sales and partnerships. Founded in 1980 and headquartered in Thousand Oaks, California (NASDAQ: AMGN), Amgen is a component of the Dow Jones Industrial Average and completed the acquisition of Horizon Therapeutics in 2023.