12 days ago
Base Salary
$165k - $185k/yr
Responsibilities
- Develop comprehensive test strategies and detailed test cases from product requirements.
- Design, implement, and maintain manual and automated tests across unit, component, integration, and end-to-end levels.
- Build scalable automated test frameworks and AI-agent evaluation tooling.
- Develop UI and API test automation using Selenium, pytest, and Playwright.
- Conduct performance and load testing using tools such as locust.
- Create test reports with reporting tools such as Allure.
- Report regressions, provide feedback on design documents, and improve testability.
- Evaluate agent task completion, tool selection, tool-call accuracy, failure recovery, latency, token usage, and cost.
- Guide evaluation practices, mentor engineers, integrate evaluations into CI/CD, and convert production failures into regression coverage.
- Collaborate with engineering, product management, data science, and UX in an autonomous cross-functional team.
Requirements
- Experience evaluating AI agents, including multi-step task completion, tool selection, tool-call accuracy, and failure recovery.
- Working knowledge of agent frameworks, preferably LangChain, including context handling, tool execution, and agent state.
- Hands-on experience with LangSmith or similar platforms for evaluation datasets, custom evaluators, experiments, and comparisons.
- Experience with agent observability using traces and telemetry to investigate quality issues, latency, token usage, and cost.
- Strong Python software engineering skills for building evaluation tooling, metrics pipelines, and reporting systems.
- Applied statistics and data analysis skills for assessing variability, comparing experiments, and distinguishing meaningful improvements from noise.
- Experience developing evaluation datasets and scoring methods using deterministic checks, human feedback, and LLM-based judges.
- Ability to contribute at a technical lead level and mentor engineers.
- JavaScript or TypeScript experience is preferred.
- Modeling experience is a plus.
Benefits
- Base salary range of $165,000-$185,000 in the United States, plus eligibility for a target annual bonus of 15% of base salary.
- Monthly Connectivity Plus stipend of $150 for remote work-related expenses.
- Annual $200 home office reimbursement.
- Medical, dental, and vision coverage.
- 401(k) with company match.
- Paid parental leave, caregiver leave, and flexible time off.
- Mental health support and wellness reimbursement.
- Career development and education assistance.
- The role involves access to software and technology subject to U.S. export controls, and employment is contingent on the applicant’s capacity to comply with those controls.
Tech Stack
Categories
About Alteryx
Alteryx builds an analytics automation platform used by analysts, data engineers, and data scientists to prepare, blend, and analyze data with code-optional tools. Its products, including Alteryx Designer plus cloud and server offerings, are sold primarily via subscriptions for on-premises and SaaS deployments across enterprises. Founded in 1997 and headquartered in Irvine, California, Alteryx is privately held under Clearlake Capital.