17 days ago
Hyderābād, IndiaMid Level / Senior
Responsibilities
- Develop, test, maintain, and troubleshoot scalable data pipelines and ETL/ELT workflows using Databricks, PySpark, and Python.
- Ingest, transform, and process structured and semi-structured data from multiple sources.
- Perform data cleansing, validation, quality checks, and performance optimization for Spark jobs and Databricks notebooks.
- Collaborate with data engineers, analysts, data scientists, and cross-functional stakeholders to deliver reliable datasets.
- Support AI/ML data preparation, including feature engineering, dataset creation, and model input preparation.
- Monitor scheduled jobs and workflows and assist with resolving pipeline failures, data issues, and performance bottlenecks.
- Create and maintain documentation for pipelines, workflows, data definitions, and processes.
- Follow practices for version control, code quality, testing, and deployment.
Requirements
- 3-5 years of experience with a bachelor’s degree in computer science, data engineering, information systems, engineering, mathematics, or a related field, or equivalent practical experience.
- Hands-on experience with Python, PySpark, Databricks, Delta Lake, SQL, and scalable ETL/ELT pipelines.
- Experience with Databricks notebooks, clusters, jobs, workflows, Delta tables, and performance optimization.
- Knowledge of structured and semi-structured formats including CSV, JSON, Parquet, and Delta.
- Understanding of data pipelines, data lakes, data warehouses, batch processing, data quality, and lakehouse architecture.
- Basic understanding of AI and machine learning concepts and experience supporting data preparation or feature engineering for AI/ML use cases.
- Familiarity with cloud-based data platforms, Git or other version control tools, and data engineering best practices.
- Preferred qualifications include experience with Unity Catalog, workflow orchestration, MLflow, scikit-learn, Tableau, Power BI, or similar tools, plus knowledge of data governance, security, and access control.
- Strong analytical, problem-solving, troubleshooting, communication, and collaboration skills.
Tech Stack
Categories
Data Engineering
About Amgen
Amgen is a public biotechnology company that discovers, develops, manufactures, and sells biologic and biosimilar therapeutics for patients with serious diseases. Its portfolio and pipeline span oncology, cardiovascular disease, osteoporosis, inflammation, and rare diseases, marketed globally through direct sales and partnerships. Founded in 1980 and headquartered in Thousand Oaks, California (NASDAQ: AMGN), Amgen is a component of the Dow Jones Industrial Average and completed the acquisition of Horizon Therapeutics in 2023.