3 days ago
Base Salary
$144k - $194k/yr
Responsibilities
- Build reusable automated pipelines for evaluating models, prompts, tools, and agent architectures.
- Provision realistic environments, run deployed agents, grade results, and compare candidates against baselines.
- Design A/B testing, shadow evaluation, online evaluation, canary release, and controlled experiment systems.
- Implement treatment assignment, exposure logging, guardrail metrics, stopping criteria, rollback, and evaluation dashboards.
- Define metrics for quality, safety, latency, cost, and reliability and produce post-launch monitoring readouts.
- Package model, prompt, tool, planning, memory, and agent-coordination changes behind stable interfaces.
- Own end-to-end science-side integration, including APIs, data contracts, versioning, telemetry, release tests, scaling, durability, tenancy, and failure recovery.
Requirements
- At least 3 years of non-internship professional software development experience.
- At least 2 years of experience designing or architecting new and existing systems for reliability and scaling.
- At least 1 year of software development engineer or related occupational experience.
- At least 1 year of experience developing large-scale, multi-tiered, multi-threaded, embedded, or distributed software applications, tools, systems, or services using C#, C++, Java, or Perl.
- At least 1 year of Object Oriented Design experience.
- Bachelor’s degree or foreign equivalent in Computer Science, Engineering, Mathematics, or a related field.
- Experience programming in at least one software programming language.
- Preferred: at least 3 years of full software development life-cycle experience, including coding standards, code reviews, source control management, build processes, testing, and operations.
- Preferred: bachelor’s degree in computer science or equivalent.
- Experience with ML or evaluation infrastructure, experiment design, A/B testing, or online evaluation.
- Experience with agent systems, LLM evaluation, or ML pipelines is preferred.
Benefits
- Amazon offers health insurance, including medical, dental, vision, prescription, Basic Life & AD&D insurance, supplemental life plan options, EAP, mental health support, a medical advice line, flexible spending accounts, and adoption and surrogacy reimbursement coverage.
- Benefits include 401(k) matching, paid time off, and parental leave.
- Compensation may include sign-on payments and restricted stock units.
- Work location: Seattle, Washington, USA.
Categories
About Amazon
Amazon builds and operates a global e-commerce marketplace, logistics network, and consumer devices, and provides cloud computing via AWS for businesses and developers. The company earns revenue from online retail, third‑party seller services, subscriptions like Prime, advertising, and AWS usage. Founded in 1994 and headquartered in Seattle, it is publicly traded on NASDAQ (AMZN) and serves customers in dozens of countries.