Handshake

Senior Applied AI Engineer

Handshake
Apply
24 hours ago

Base Salary

$220k - $260k/yr

Responsibilities

  • Build a scalable evaluation platform that turns ambiguous data-quality needs into measurable outcomes and reusable products.
  • Design and deploy production agentic workflows for generating, reviewing, investigating, and improving data with tool use, orchestration, human oversight, and failure recovery.
  • Develop graders, programmatic verifiers, reward signals, evaluation datasets, benchmarks, and agent harnesses across LLMs, agents, multimodal models, and reinforcement-learning environments.
  • Analyze traces, diagnose failure modes, run experiments, and improve system performance based on meaningful evaluation metrics.
  • Own projects from concept through production rollout, set technical direction, mentor engineers, and collaborate with product, design, operations, research, and customers.

Requirements

  • 5+ years of engineering experience.
  • Substantial experience shipping and operating high-quality software and making product decisions across the stack.
  • Production experience building agentic workflows, including tool calling, context management, orchestration, retrieval, human review, and reliability tradeoffs.
  • Strong Python skills and ability to build scalable services, APIs, and data pipelines with disciplined testing, debugging, and system design.
  • Experience selecting meaningful metrics, building representative test sets, assessing graders, and distinguishing real improvements from noise.
  • Fluency in modern AI models, agent harnesses, benchmarks, evaluation techniques, and the effects of training data and post-training methods on model behavior.
  • Strong technical judgment, ownership, communication, and collaboration across engineering, research, operations, domain experts, and customers.
  • Preferred experience with reinforcement learning, alignment, model behavior, synthetic data, human-in-the-loop systems, LLM or agent benchmarks, reward models, evaluation methodologies, annotation systems, or data-quality frameworks.
  • Demonstrated technical ability through shipped products, open-source contributions, benchmarks, datasets, or other substantive work.

Benefits

  • Equity in a fast-growing company.
  • 401(k) match, competitive compensation, and financial coaching.
  • Medical, dental, and vision coverage for full-time US employees.
  • Mental health support and a $500 wellness stipend.
  • Paid parental leave, fertility benefits, and parental coaching.
  • $2,000 annual learning stipend.
  • Flexible PTO, 15 holidays, and 2 flex days.
  • Internet and commuting support, plus free lunch and gym access at the San Francisco office.
  • Team outings and referral bonuses.

Tech Stack

Categories

Handshake

About Handshake

10,000+ employees

Handshake builds a career network and recruiting platform that connects college students and recent grads with employers, sold as SaaS to universities and subscriptions/solutions to employers. Founded in 2014 and headquartered in San Francisco, it serves 1,600+ educational institutions and over 1 million employers. The company also operates Handshake AI, which partners with frontier AI labs on human data collection and evaluations for model training and post-training workflows.

Contact me