product engineer, agent
Judgment Labs28 days ago
Responsibilities
- Build the Judgment Agent for large-scale investigations across production traces, including parallel investigators, failure modes, tool errors, regressions, and drift.
- Build verification infrastructure with hosted simulated environments, stateful agent evaluations, trajectory replay, and monitoring for unintended behavior changes.
- Design interfaces that make long agent traces, tool calls, decisions, and failures understandable to engineers.
- Design swarm experiences for monitoring, redirecting, and consuming findings from parallel investigations.
- Build workflows that turn production trajectories into datasets, judges, and regression checks.
- Develop platform capabilities including workspaces, roles, permissions, billing, usage, and limits.
- Build an SDK and terminal-first experience for integrating Judgment into Claude Code, Codex, and OpenCode sessions.
- Own problems end-to-end by talking to customers, defining what to build, implementing it, and iterating until it is effective.
Requirements
- Experience building and scaling end-to-end production systems from the data layer through the UI.
- Strong technical problem-solving skills in fast-changing, ambiguous environments.
- Hands-on experience building with LLMs or agents, or the ability to develop that experience quickly.
- Comfort working directly with customers to understand needs and solve real-world problems.
- Excellent communication skills across technical and non-technical audiences.
- A high-agency builder and tinkerer mindset with creative approaches to overcoming obstacles and shipping.
Categories
AI ApplicationsProduct Engineering
About Judgment Labs
Judgment Labs builds learning and evaluation infrastructure for AI agents, helping engineering teams monitor behavior, surface failure modes, and improve performance from real production traces. Its platform and SDK provide continuous testing, rubrics, and large-scale investigations across tool errors, regressions, and drift. Founded in 2025 and headquartered in San Francisco, the privately held company sells B2B software for teams deploying long‑trajectory agents in development and production environments.