about 18 hours ago
Waltham, MA, USA
Senior
H1B Sponsor
Base Salary
$140k - $220k/yr
Responsibilities
- Own and drive the design and implementation of large-scale data pipelines.
- Architect resilient ETL/ELT pipelines handling massive volumes of data.
- Proactively identify complex technical challenges and propose solutions.
- Build streaming and batch processing systems for data flows.
- Establish data quality frameworks to ensure data accuracy and consistency.
- Define and implement observability and monitoring for pipeline health.
- Drive technical design decisions and guide implementations.
- Mentor and elevate other engineers through code reviews and knowledge sharing.
- Collaborate with product, platform, and data science teams to deliver features.
- Influence technical direction across the Contributory Network initiative.
Requirements
- 5+ years of professional software engineering experience focused on data engineering.
- Proven track record of building and operating production data pipelines at scale.
- Deep experience with Python and/or Java.
- Hands-on expertise with data processing technologies like Apache Beam and Spark.
- Strong experience with streaming systems such as Apache Kafka.
- Experience with cloud platforms, preferably GCP.
- Solid understanding of data modeling and data quality management.
- Experience designing and operating large-scale ETL/ELT pipelines.
Tech Stack
Apache AirflowApache BeamApache KafkaApache SparkGoogle BigQueryGoogle Cloud PlatformJavaKubernetesPythonSnowflakeTerraform
Categories
AI & MLData Engineering