5 hours ago
Remote, WorldwideSenior
Base Salary
$166k - $416k/yr
Responsibilities
- Design reinforcement learning environments with Python, Docker, state spaces, tool integrations, and scoring mechanisms.
- Build robust verifiers and reward functions that resist agent gaming.
- Translate ERP, CRM, finance, and operations workflows into measurable evaluation scenarios.
- Integrate tools and REST APIs and build test harnesses using pytest or similar tools.
- Lead agent demonstration and computer-use programs and establish trace-collection quality standards.
- Grade and repair agent trajectories and mentor contributors on evaluation practices.
Requirements
- At least 5 years of experience in software engineering, QA automation, machine learning, reinforcement learning, or business systems and operations paired with hands-on agent or reinforcement learning data work.
- Strong Python skills and experience building tools or services used by others.
- Experience designing environments, verifiers, reward functions, or test harnesses for evaluating software or agent behavior.
- Experience with pytest, Playwright, Selenium, or Gymnasium-style environments.
- Hands-on experience with agent or reinforcement learning data, including demonstrations, trajectory grading, trace collection, or computer-use demos.
- Experience with REST APIs and Docker.
- Experience leading or reviewing data, evaluation, or demonstration work is valuable.
- Expertise modeling workflows in business systems such as Salesforce or ERP platforms is valuable.
- A degree in computer science, engineering, or a related field is relevant.
Benefits
- Remote work arrangement.
- Projects typically run from one week to several months.
- Compensation is $80 to $200 per hour depending on project, field, and level, paid based on hours worked.