20 days ago
Base Salary
$200k - $280k/yr
Responsibilities
- Build evalsets and benchmarks from expert-curated, production, synthetic, and historical failure data.
- Scale evaluation platforms, experiment pipelines, agent observability, trace capture and replay, and dashboards.
- Build and calibrate LLM verifiers for correctness, groundedness, citation quality, and task success.
- Run evaluations from CI and pre-release suites through sampled production traffic and report results with confidence intervals.
- Build enterprise simulation environments with simulated users and realistic company data for multi-step agent testing.
- Diagnose failures and translate evaluation results into product and engineering fixes while balancing quality, latency, and cost.
- Partner with R&D, product, and go-to-market teams to apply evaluation findings to customer outcomes.
Requirements
- 5+ years of software engineering experience with strong Go and/or Python and backend fundamentals.
- Experience with APIs, data modeling, testing, distributed pipelines, and reliable infrastructure at scale.
- Experience building datasets and benchmarks for LLM use cases, RAG, or agents, including evalset curation and synthetic data generation.
- Familiarity with deterministic metrics, LLM-as-a-judge, pointwise and pairwise grading, human labeling, and judge calibration.
- Experience integrating evaluation or experiment infrastructure with CI/CD and observability tooling such as OpenTelemetry.
- Strong analytical skills, including investigating traces and failures, writing SQL, and using data and metrics to explain quality changes.
- TypeScript experience is a plus.
- Experience with simulation environments and/or model post-training techniques is a plus.
Benefits
- Hybrid work arrangement requiring four days per week in the Mountain View or San Francisco office.
- Medical, vision, and dental coverage.
- Generous time-off policy and access to a 401k plan.
- Home office improvement stipend.
- Annual education and wellness stipends.
- Regular company events and healthy lunches daily.
- Certain roles may be eligible for variable compensation, equity, and benefits.
Tech Stack
Categories
About Glean
Glean.ai is Intelligent AP. We leverage untapped invoice data to surface relevant context, actionable insights, and savings recommendations that power great spend decisions.