Lead Data Platform Engineer - 11623
Coupa Software14 days ago
Base Salary
$125k - $174k/yr
Responsibilities
- Manage end-to-end data pipeline (ETL jobs) within agreed SLAs.
- Manage AWS core and big data services such as S3, IAM, EMR, and Redshift.
- Run applications in containers using ECS and Docker.
- Lead Day 2 operational lifecycle for ML and GenAI infrastructure.
- Collaborate with product development and engineering teams to create AI-driven features.
- Automate platform maintenance and standardize infrastructure configurations.
- Manage AWS infrastructure using code with tools like Terraform and Chef.
- Administer applications running in a Linux operating system.
- Enable application and system monitoring for better observability.
- Provide application and infrastructure support for ETL jobs and data pipelines.
- Participate in design reviews, code reviews, and troubleshooting incidents.
- Handle multiple priorities in a high-pressure environment.
Requirements
- Bachelor's Degree and at least 8+ years of experience managing Big Data technologies.
- Sound knowledge and experience in Linux administration and troubleshooting.
- 5+ years of experience in managing cloud infrastructure and platforms like AWS and Azure.
- Familiarity with the generative AI landscape and strong interest in AI technologies.
- Strong expertise in MLOps and production-grade LLM operations.
- Experience with Bash or Python scripting.
- Experience with containerization, Amazon ECS, and Azure AKS.
- Familiarity with tools like Chef, Ansible, Jenkins, and Rundeck.
- Experience with source control systems such as Git.
- Experience with Infrastructure as Code products like Terraform.
- Good understanding of DNS and Load balancers setup and troubleshooting.
- Experience in Big Data platforms and managing Business Intelligence tools.
- Knowledge in ApacheSpark architecture and troubleshooting Java applications.
- Basic understanding of MySQL Server and general database knowledge.
- Excellent written and verbal communication skills.