2 days ago
Base Salary
$320k - $485k/yr
Responsibilities
- Design and run experiments to improve evaluation quality, test-data representativeness, grading accuracy, and evaluation coverage.
- Build and own an evaluation harness for an agentic investigation system, including metrics, test cases, grading approaches, and end-to-end performance measurement.
- Construct reinforcement-learning environments to improve Claude's safety investigation capabilities.
- Create evaluation datasets representing real-world misuse using real traffic patterns and synthetic generation.
- Collaborate with Policy and Enforcement to translate observed harm patterns into measurable evaluations.
- Ship evaluation methods into regression and release pipelines used during model training, agent changes, prompt updates, model upgrades, and post-launch operations.
- Build tooling that enables policy experts to author, run, and iterate on evaluations without engineering support.
- Surface evaluation findings to research and training teams to drive model improvements.
Requirements
- Experience building and maintaining data pipelines.
- Experience working with LLMs and understanding their capabilities and failure modes, especially agentic systems with tool use and multi-step reasoning.
- Strong data analysis skills and the ability to draw reliable insights from large datasets.
- Ability to move between research prototyping and production-quality code.
- Ability to translate ambiguous problems into concrete, testable experiments.
- Strong interest in AI safety and meaningful societal impact.
- Preferred: 8+ years of industry software engineering or ML engineering experience.
- Preferred: expertise with LLM or agent evaluation frameworks, benchmarks, or automated grading systems.
- Preferred: extensive experience in trust and safety, content moderation, abuse detection, red teaming, adversarial testing, or jailbreak research.
- Preferred: experience with synthetic data generation, data augmentation, distributed systems, large-scale data processing, prompt engineering, or LLM-powered applications.
- Bachelor's degree or an equivalent combination of education, training, and/or experience in a relevant field.
Benefits
- Annual base salary range of $320,000-$485,000 USD.
- Hybrid policy requiring staff to work from an office at least 25% of the time, with some roles requiring more office time.
- Visa sponsorship is offered with reasonable efforts made after an offer, supported by an immigration lawyer.
- Competitive compensation and benefits, optional equity donation matching, generous vacation and parental leave, flexible working hours, and office collaboration space.
Categories
About Anthropic
Anthropic builds large language models and the Claude AI assistant for developers and enterprises, offered via API access and enterprise plans. Founded in 2021 and headquartered in San Francisco, it distributes Claude through its own platform and via partners such as Amazon Bedrock and Google Cloud’s Vertex AI. Its work emphasizes model reliability, interpretability, and practical tooling for tasks like coding assistance, analysis, and customer support automation.