2 months ago
London, United KingdomSenior
Responsibilities
- Design and build a unified evaluation system for agentic and algorithmic solutions across task success, quality, robustness, motifs, agent topologies, and decomposition strategies.
- Mine real commits and traces and create task suites covering code search, code editing and repair, repository summarization, and tool use.
- Implement sandboxed execution grading and maintain reproducible, contamination-controlled, stable benchmarks and baselines.
- Compare agentic and algorithmic approaches based on resolved-task quality and transparently account for added steps, latency, and cost.
- Lead external benchmark co-publications and ensure results withstand peer and customer review.
- Use evaluation results to inform shipping decisions, customer proof points, and company-wide quality claims.
Requirements
- PhD in computer science, machine learning, or a related field, or an equivalent research track record.
- Authorship or co-authorship of a benchmark or evaluation paper at a recognized venue such as NeurIPS Datasets and Benchmarks, ICML, ICLR, or ACL.
- Working understanding of contamination, benchmark overfitting, weak baselines, underpowered comparisons, and irreproducible results in LLM and agent evaluation.
- Strong Python engineering skills and comfort with sandboxed and distributed execution and CI.
- Hands-on experience building or rigorously evaluating agentic or multi-step LLM systems.
Benefits
- Competitive salary determined by skills and experience.
- Equity and ownership.
- Private healthcare.
- Visa sponsorship and relocation benefits.
- In-person work at the London office with provided tools, workspace, and setup.
Tech Stack
Categories
About Callosum
Callosum builds AI infrastructure that unifies heterogeneous compute—software and systems to orchestrate diverse accelerators and the physical clusters they run on—for organizations training and serving models. The privately held company was founded in 2025 and is headquartered in London. It focuses on designing, deploying, and operating mixed-hardware compute fleets across sites and vendors, and raised a seed round in 2026.