2 days ago
London, United KingdomStaff+
Responsibilities
- Lead the technical strategy and implementation of agent evaluation at PhysicsX.
- Build reusable evaluation frameworks, reference implementations, and standardized workflows on existing tooling.
- Help product, delivery, and customer teams establish robust evaluation suites and turn successful practices into repeatable guidance.
- Develop datasets and evaluators using deterministic checks, domain-expert judgment, model-based grading, and human review.
- Connect offline evaluation, regression testing, and production telemetry to detect drift and learn from real-world use.
- Define cross-product scorecards and shared benchmarks while leaving product teams accountable for their own agents.
- Create and publish open benchmarks for agents performing engineering and manufacturing workflows.
- Own the integrity of cross-product, production-agent, and public benchmarks, collaborating with Research and Post-Training on model and agent evaluation.
- Build organizational capability through reviews, examples, training, and direct collaboration.
- Help determine how the evaluation capability should grow and shape the future team.
Requirements
- Experience building and operating production evaluation systems for agents or LLM-powered products.
- Ability to translate ambiguous product goals into trusted datasets, evaluators, experiments, and decision criteria.
- Strong understanding of probabilistic-system failure modes and appropriate statistical judgment.
- Knowledge of when to use deterministic evaluators, domain experts, human review, and LLM-as-a-judge, including their limitations.
- Strong software engineering skills with experience building usable frameworks and production-quality tooling.
- Experience helping multiple teams adopt technical practices through hands-on delivery, standards, and coaching.
- Ability to work directly with customers and domain experts to encode workflow correctness without losing nuance.
- Knowledge of evaluation for multi-step agents, tool calls, structured outputs, and long-running workflows.
- Understanding of trace-based observability and OpenTelemetry concepts.
- Knowledge of dataset design, sampling, experiment design, regression detection, and reproducibility.
- Experience with CI and release gates for non-deterministic systems.
- Clear technical writing and communication across product, research, engineering, delivery, and customer teams.
- Strong product judgment about which measurements should drive decisions.
- Preferred experience with Pydantic Evals, Pydantic Logfire, or comparable code-first evaluation and observability tools.
- Preferred experience evaluating agents in engineering, simulation, design, manufacturing, or other expert domains.
- Preferred experience creating public benchmarks or open-source evaluation tooling.
- Familiarity with post-training, synthetic data, reward signals, or model selection is beneficial.
- Experience in regulated, customer-operated, or air-gapped environments is beneficial.
Categories
About PhysicsX
PhysicsX builds an AI-driven simulation software stack that enables high-fidelity, multi-physics modeling and optimization for engineering and manufacturing teams. The company sells software and delivery services to enterprises in aerospace and defense, automotive, semiconductors, materials, and energy. It is privately held and headquartered in London, with offices in London and New York.