25 days ago
Base Salary
$225k - $250k/yr
Responsibilities
- Architect and build scalable infrastructure for generating, transforming, validating, and delivering large-scale coding datasets.
- Design end-to-end evaluation systems covering automated grading, benchmarking, human-in-the-loop review, and quality assurance workflows.
- Lead technical design for developer-facing tooling and integrations with GitHub, CI/CD systems, coding agents, and containerized execution environments.
- Build reliable backend services and APIs supporting dataset generation, evaluation pipelines, and experiment infrastructure.
- Drive architectural decisions for distributed systems, workflow orchestration, execution environments, and data quality.
- Partner with ML researchers to translate evolving evaluation methodologies into scalable engineering systems.
- Improve platform reliability, observability, and performance through monitoring, debugging, and operational excellence.
- Mentor engineers through technical design reviews, code reviews, and architectural guidance.
- Standardize reusable infrastructure, tooling, and evaluation frameworks to accelerate future model development.
Requirements
- At least 6 years of professional software engineering experience building backend systems, data infrastructure, or distributed platforms.
- Strong programming skills in Python, TypeScript, Java, or similar languages.
- Experience designing and operating large-scale data pipelines, distributed systems, or workflow orchestration platforms.
- Strong system design skills, including architectural decisions involving scalability, reliability, observability, and maintainability.
- Experience building cloud-native systems using AWS, GCP, or similar cloud platforms.
- Familiarity with containerized execution environments such as Docker and Kubernetes and with distributed job processing.
- Strong understanding of relational or NoSQL databases, data modeling, and storage systems.
- Ability to translate evolving research or product requirements into scalable engineering solutions.
- Excellent communication skills and effective collaboration with researchers, product managers, and engineering teams.
- Experience mentoring engineers and leading technical projects from design through production.
- Preferred experience includes ML data infrastructure, evaluation frameworks, benchmarking systems, or dataset generation pipelines.
- Preferred experience includes coding agents, AI-assisted software development tools, developer productivity platforms, GitHub APIs, CI/CD platforms, or code execution environments.
- Preferred familiarity with Airflow, Temporal, Dagster, LLM evaluation, coding benchmarks, or agentic software engineering systems.
- Background in infrastructure, platform engineering, or developer tooling in high-growth environments is preferred.
Benefits
- Equity in a fast-growing company, 401(k) match, competitive compensation, and financial coaching.
- Paid parental leave, fertility benefits, and parental coaching.
- Medical, dental, vision, mental health support, and a $500 wellness stipend.
- A $2,000 learning stipend and ongoing development opportunities.
- Commuting support, free lunch, and gym access at the San Francisco office.
- Flexible PTO, 15 holidays, and 2 flex days.
- Team outings and referral bonuses.
- Full-time role based in San Francisco, California, with benefits listed for U.S. employees.