3 days ago
Remote, ArgentinaSenior
Responsibilities
- Maintain and improve evaluation, RL, and annotation platforms, including bug fixes, technical-debt reduction, testing, and documentation.
- Design and ship features across databases, backend services, APIs, and internal tools.
- Scale distributed systems for larger workloads with predictable performance and cost.
- Build RL environments and the infrastructure for sandboxes, environment images, rollouts, and reward computation.
- Operate reliable and secure production systems through monitoring, alerting, incident response, access control, untrusted-code isolation, and client-data protection.
- Build tooling for task authors, reviewers, and project leads and support rapid-turnaround data samples.
- Collect high-throughput rollouts with parallel episodes, timeouts, resource limits, logs, and trajectories.
- Build evaluation harnesses, verifiers, graders, and reward functions for LLMs and LLM agents.
- Support contributor and reviewer workflows, quality control, calibration, and scaled human-data processes.
Requirements
- At least 5 years of experience in software, data, or ML engineering, with recent work in AI/ML data, model evaluation, annotation, or developer tooling.
- Strong backend engineering experience in Python, including clean, tested, and maintainable code.
- Experience designing APIs, data models, relational database schemas, migrations, and query-performance improvements using Postgres or similar systems.
- Enough TypeScript and React experience to build and maintain internal tools.
- Experience designing distributed systems with long-running jobs, orchestration, workflow engines, queues, retries, idempotency, and failure recovery.
- Experience with containers, container orchestration, and at least one major cloud provider.
- Experience with observability, including logging, metrics, tracing, and alerting, plus security fundamentals such as secrets management, isolation, and access control.
- Hands-on experience building RL environments for LLM agents, operating isolated sandboxes at scale, building environment images, collecting rollouts, and feeding rewards into RL post-training pipelines.
- Hands-on experience building or running LLM or LLM-agent evaluation harnesses or benchmarks and understanding evaluation concepts such as pass rates, pass@k, headroom, discrimination, and reward hacking.
- Experience with annotation, labeling, or human-data tooling and with scaling manual processes into measured workflows.
- A track record of running and maintaining production systems, fluent English, strong written communication, independent execution, and comfort working in ambiguity.
- Preferred qualifications include contributions to open-source evaluation or RL frameworks and experience building tools for a large distributed contributor workforce.
- Daily careful use of AI coding agents and rigorous review of their output.
Benefits
- 100% remote work arrangement.
- Opportunity to work on frontier AI training and evaluation data platforms.
- Core ownership and broad technical scope on a small, fast-moving team.