5 hours ago
Base Salary
$215k - $300k/yr
Responsibilities
- Build Profound’s evaluation system with golden datasets, scorers, LLM judges, programmatic verifiers, regression alerts, and dashboards.
- Create evaluations for brand extraction, visibility scoring, sentiment, FactCheck, and citation classification pipelines.
- Define and measure AI Marketer quality across multi-step tool use, analytical correctness, and content quality.
- Study answer-engine behavior across runs, platforms, regions, and personas using sound sampling and statistical methods.
- Collect customer signals, cluster failure modes, and convert regressions into permanent guardrails.
- Build debugging tools for model and agent behavior and create labeling workflows and annotation guidelines.
- Integrate evaluations into CI, launch reviews, and model-selection processes.
Requirements
- At least 3 years of experience building evaluation or measurement systems, including AI evaluations, experimentation, ranking, or search quality.
- Strong Python and data skills with a record of shipping production systems.
- Deep familiarity with LLM-as-judge evaluation, benchmark design, agent harnesses, calibration, and significance testing.
- Strong experimental-design skills and the ability to separate signal from noise in non-deterministic systems.
- Current understanding of model and agent behavior and related research.
- Evidence of shipped benchmarks, datasets, open-source tools, or publications.
- Ability to work in a fast-paced startup environment with rapid iteration.
- Preferred: experience with search, retrieval, citation, or fact-verification systems.
- Preferred: publications at NeurIPS, ICML, ICLR, ACL, or COLM.
Benefits
- On-site work in the New York City or San Francisco office.
- Base salary range of $215,000 to $300,000, plus equity and a full range of benefits and perks.
Tech Stack
Categories
About Profound
Profound builds a SaaS marketing intelligence platform that helps enterprise brands measure and improve how they appear in AI-driven discovery across tools like ChatGPT and Perplexity. The product delivers real-time visibility analytics, competitive intelligence, and optimization insights for search-like AI channels. The company is privately held, headquartered in New York, and is Series C–funded.