4 hours ago
Responsibilities
- Design and ship features spanning data models, APIs, background workers, and UI components.
- Build and improve evaluation workflows for coding agents and language models, including sandboxed trials, scoring, and result aggregation.
- Develop benchmark quality checks using verifiers, rubric-based grading, and LLM-as-a-judge evaluation.
- Integrate new models, agents, and benchmarks through generalizable systems rather than special-case implementations.
- Improve system performance and reliability through efficient queries, safe migrations, and concurrency controls.
- Review pull requests and maintain consistent engineering standards across the codebase.
Requirements
- Typically 5+ years of experience building and operating production backend systems, including ownership of a service from design through on-call.
- Strong Python experience with Django or a similar framework, PostgreSQL, and a task queue such as Celery.
- Hands-on experience with LLM evaluation, including benchmark construction, agent-trial scoring, and result aggregation and comparison.
- Ability to assess benchmark task quality, verifier weaknesses, leaked answers, ambiguous instructions, and misleading checks.
- Experience owning loosely defined work through design, migrations, tests, edge cases, and production delivery.
- Comfort working with TypeScript and React when frontend changes are required.
- Clear technical writing and ability to make pragmatic architecture and implementation tradeoffs.
- Preferred experience includes Harbor, Terminal-Bench, SWE-bench, Inspect, sandboxed or containerized agents, Docker, Kubernetes, Modal, Daytona, distributed job scheduling, concurrency controls, database locking, coding agents, MCP, and open-source evaluation or infrastructure projects.
Benefits
- Work at the frontier of AI and help leading AI labs improve models through datasets, reinforcement learning environments, and benchmarks.
- Contribute to AI research and potentially showcase work at conferences such as ICLR, ICML, and NeurIPS.
- Apply frontier AI research to enterprise challenges and collaborate with experienced colleagues from major technology companies.
- Work in a startup-paced environment with substantial ownership and impact.
- The posting does not state a specific work arrangement, contract duration, or compensation details.
Tech Stack
Categories
About Turing
Turing builds AI data pipelines and systems for enterprises and AI labs. It creates large-scale datasets and reinforcement-learning environments to train and evaluate models, and deploys agentic AI applications in domains like software engineering and enterprise workflows. The company also operates an AI-powered talent cloud to match and manage developers and AI trainers. Founded in 2018 and headquartered in San Francisco, it is privately held and serves Fortune 500 customers.