24 hours ago
Base Salary
$220k - $260k/yr
Responsibilities
- Build a scalable evaluation platform that turns ambiguous data-quality needs into measurable outcomes and reusable products.
- Design and deploy production agentic workflows for generating, reviewing, investigating, and improving data with tool use, orchestration, human oversight, and failure recovery.
- Develop graders, programmatic verifiers, reward signals, evaluation datasets, benchmarks, and agent harnesses across LLMs, agents, multimodal models, and reinforcement-learning environments.
- Analyze traces, diagnose failure modes, run experiments, and improve system performance based on meaningful evaluation metrics.
- Own projects from concept through production rollout, set technical direction, mentor engineers, and collaborate with product, design, operations, research, and customers.
Requirements
- 5+ years of engineering experience.
- Substantial experience shipping and operating high-quality software and making product decisions across the stack.
- Production experience building agentic workflows, including tool calling, context management, orchestration, retrieval, human review, and reliability tradeoffs.
- Strong Python skills and ability to build scalable services, APIs, and data pipelines with disciplined testing, debugging, and system design.
- Experience selecting meaningful metrics, building representative test sets, assessing graders, and distinguishing real improvements from noise.
- Fluency in modern AI models, agent harnesses, benchmarks, evaluation techniques, and the effects of training data and post-training methods on model behavior.
- Strong technical judgment, ownership, communication, and collaboration across engineering, research, operations, domain experts, and customers.
- Preferred experience with reinforcement learning, alignment, model behavior, synthetic data, human-in-the-loop systems, LLM or agent benchmarks, reward models, evaluation methodologies, annotation systems, or data-quality frameworks.
- Demonstrated technical ability through shipped products, open-source contributions, benchmarks, datasets, or other substantive work.
Benefits
- Equity in a fast-growing company.
- 401(k) match, competitive compensation, and financial coaching.
- Medical, dental, and vision coverage for full-time US employees.
- Mental health support and a $500 wellness stipend.
- Paid parental leave, fertility benefits, and parental coaching.
- $2,000 annual learning stipend.
- Flexible PTO, 15 holidays, and 2 flex days.
- Internet and commuting support, plus free lunch and gym access at the San Francisco office.
- Team outings and referral bonuses.
Tech Stack
Categories
About Handshake
Handshake builds a career network and recruiting platform that connects college students and recent grads with employers, sold as SaaS to universities and subscriptions/solutions to employers. Founded in 2014 and headquartered in San Francisco, it serves 1,600+ educational institutions and over 1 million employers. The company also operates Handshake AI, which partners with frontier AI labs on human data collection and evaluations for model training and post-training workflows.
