Backend Engineer
Bespoke Labs3 months ago
Responsibilities
- Design and own the sandboxing and execution layer for RL environments, including isolation and snapshot, restore, inspection, and branching capabilities.
- Build systems that detect rollout failures, reward hacks, infrastructure faults, and fairness issues, then revert, patch, and continue execution.
- Extend execution to long-horizon and multi-node environments operating across many tools and services.
- Optimize throughput, latency, cost per rollout, scheduling, resource utilization, and container startup and teardown performance.
- Build observability for thousands of concurrent, long-running rollouts.
- Develop and maintain frameworks for specifying, packaging, and deploying RL environments.
- Create tools for debugging failures across long agent traces and scale prototypes into reproducible production systems.
- Document the platform and enable internal teams and external users to build on it.
Requirements
- Strong track record building production systems or research infrastructure at scale, including distributed systems, execution engines, or container and sandboxing infrastructure.
- Deep familiarity with containers, isolation, namespaces, cgroups, virtual machines, filesystems, process management, and state management.
- Experience with profiling, scheduling, resource utilization, performance optimization, and cost reduction at scale.
- Proficiency with GCP, AWS, and distributed computing.
- Strong engineering fundamentals and a systematic approach to testing, validation, and reliability.
- Strong Python skills; Rust, Go, or C++ experience is a plus.
- Ability to operate in ambiguity, use modern tools such as Claude Code, and communicate with research teams and enterprise customers.
- Preferred experience with RL training or evaluation infrastructure, agent rollout execution, checkpoint or snapshot-restore systems, CRIU, distributed state management, high-throughput or low-latency execution systems, or widely used infrastructure and open-source systems.
Benefits
- Health coverage
- Opportunity to work directly with leading AI research labs
- On-site role located in Mountain View, California