Senior Data Engineer
HealthEdge5 months ago
Bengaluru, IndiaSenior
Responsibilities
- Design, build, and maintain AWS Glue and PySpark ETL jobs that ingest data from Salesforce, JIRA, Azure DevOps, Zendesk, and other source systems.
- Develop and optimize AWS Step Functions workflows with parallel execution, error handling, retries, and concurrency management.
- Implement incremental loading, schema evolution handling, data quality checks, and Apache Iceberg table maintenance on Amazon S3.
- Integrate EventBridge, SQS, SNS, Athena, CloudWatch, and IAM into data pipeline architectures.
- Troubleshoot production failures, API timeouts, out-of-memory errors, performance bottlenecks, and DPU tuning issues.
- Build, publish, maintain, and optimize Amazon QuickSight datasets, dashboards, analyses, calculated fields, filters, and refresh schedules.
- Translate business requirements and Tableau or Excel reports into accurate and usable QuickSight visuals and KPIs.
- Monitor data freshness and pipeline health, respond to alerts, participate in code reviews and sprint planning, and improve data quality and run times.
- Write clear technical documentation and communicate status proactively with stakeholders and teams across time zones.
Requirements
- 5–10+ years of professional experience in data engineering, BI development, or analytics engineering.
- Strong hands-on experience with AWS Glue, PySpark, AWS Step Functions, Amazon S3, Athena, and IAM.
- Proficiency in Python and SQL, including PySpark transformations and Athena queries.
- Experience building and maintaining ETL pipelines that ingest data from REST APIs or SaaS platforms such as Salesforce, JIRA, and Zendesk.
- Working knowledge of Amazon QuickSight, including dataset creation, calculated fields, parameters, and dashboard publishing.
- Familiarity with Apache Iceberg, Delta Lake, or Apache Hudi and with dimensional modeling, slowly changing dimensions, and incremental versus full loads.
- Ability to work independently with minimal supervision, manage priorities, communicate asynchronously, and write clear documentation in English.
- Preferred experience with AWS CDK, CloudFormation, Terraform, GitHub Actions, or CodePipeline.
- Preferred exposure to PII detection and redaction, data governance frameworks, Tableau workbook or Excel report migration, and EventBridge-driven architectures with SQS/SNS messaging.
- AWS certifications such as Data Analytics Specialty or Solutions Architect Associate are a plus.
Benefits
- Opportunity to work on a greenfield BI platform using modern cloud-native tooling.
- Collaborative small-team environment with direct and visible impact.
- Exposure to a broad technology stack spanning data engineering, analytics, and cloud infrastructure.
- Mentorship and growth opportunities within a growing data organization.
About HealthEdge
HealthEdge builds enterprise software for health insurers, including core claims administration, payment integrity, care management, and member engagement products delivered as SaaS. Its customers are health plans seeking to modernize operations and connect payers, providers, and members. The company is privately held, headquartered in Boston, and operates as part of Bain Capital’s portfolio.