2 days ago
San Jose, CA, USAMid Level
H1B sponsor
Base Salary
$180k - $450k/yr
Responsibilities
- Build and own end-to-end evaluation suites for LLM capabilities, including task specifications, grading logic, and researcher dashboards.
- Operate evaluation harnesses as production services with sandboxed execution, tool calling, multi-turn and agentic tasks, deterministic replay, and caching.
- Own data pipelines for collection, filtering, deduplication, labeling, synthetic data generation, and quality gates.
- Build and maintain RL training environments, including reward functions, simulators, verifiers, and task generators.
- Collaborate with researchers to turn hypotheses into runnable evaluations and environments and interpret results.
- Measure evaluation regressions, environment health, throughput, and cost while identifying misleading metric changes.
- Improve research-code quality through testing, versioned datasets and evaluations, and clear training, inference, and evaluation boundaries.
Requirements
- At least 3 years of software engineering experience, including 1–2 years of hands-on LLM training, evaluation, or data-infrastructure experience.
- Deep Python fluency and practical distributed-systems experience across unreliable workers and machine fleets.
- Experience building evaluation infrastructure, including tasks, graders, and supporting systems.
- Working knowledge of supervised fine-tuning and reinforcement learning for language models.
- Production-oriented engineering practices, including testing and reproducibility.
- Ability to work effectively with researchers and turn ambiguous ideas into measurable systems.
- Preferred qualifications include agentic evaluation, tool use, code execution, sandbox environments, multi-turn tasks, training frameworks, inference stacks, data-labeling operations, human-preference collection, open-source evaluation frameworks, public benchmarks, verifiers, and simulation environments.
Benefits
- Full-time position with a US base salary range of $180,000–$450,000 annually.
- Additional compensation components and benefits may be included depending on the specific role and shared if an employment offer is extended.
Categories
