10 days ago
San Francisco, CA, USA or New York, NY, USAMid Level
H1B sponsor
Base Salary
$230k - $325k/yr
Responsibilities
- Build evaluation systems with tasks, rubrics, LLM judges, programmatic verifiers, regression alerts, and self-improvement loops.
- Define and analyze AI Marketer performance across multi-step tool use, analytical correctness, and content quality.
- Partner with marketing experts and data-labeling companies to create tasks, metrics, rubrics, and grading mechanisms.
- Analyze usage, cluster failure modes, and convert them into reliable evaluations.
- Turn evaluation results and expert feedback into rewards, preference data, and training environments.
- Build tools for debugging model and agent behavior.
- Integrate evaluations into CI, launch reviews, and model selection.
Requirements
- At least 2 years of experience building evaluation or measurement systems in areas such as AI evals, experimentation, ranking, or search quality.
- Strong Python and data skills with a record of shipping production systems.
- Deep familiarity with LLM evaluation, including LLM-as-judge, benchmark design, agent harnesses, and calibration.
- Sound experimental-design skills and the ability to distinguish signal from noise in non-deterministic systems.
- Familiarity with modern post-training methods, including SFT, preference optimization, and RL.
- Strong understanding of how models and agents should behave and current AI research.
- Comfort working in a fast-paced startup with rapid iteration cycles.
Benefits
- On-site work in the company’s NYC or San Francisco office.
- Competitive total compensation including equity and a full range of benefits and perks.
Tech Stack
Categories
AI Research
About Profound
Profound builds a SaaS marketing intelligence platform that helps enterprise brands measure and improve how they appear in AI-driven discovery across tools like ChatGPT and Perplexity. The product delivers real-time visibility analytics, competitive intelligence, and optimization insights for search-like AI channels. The company is privately held, headquartered in New York, and is Series C–funded.