2 days ago
Cambridge, United KingdomSenior
Responsibilities
- Build repeatable evaluation pipelines and representative task-based datasets using MLflow and promptfoo.
- Define success criteria, establish baselines, and automate comparisons of task success, quality, latency, and cost with partner engineers.
- Instrument AI workflows with OpenTelemetry and Prometheus, and use traces, logs, and evaluation results to debug orchestration, model interactions, context management, and tool integrations.
- Implement and test improvements to AI-assisted and agentic workflows using Codex, Claude, and Microsoft Agent Framework.
- Enhance tooling and build instrumentation, evaluation, and regression-testing capabilities within established architecture.
- Integrate automated quality checks into Jenkins pipelines or GitLab CI/CD workflows.
- Package evaluation tools and workflow services with Docker and deploy and troubleshoot them in AWS or comparable cloud environments.
- Deliver assigned work with minimal guidance, raise blockers and dependencies, document findings, provide practical guidance, and conduct peer reviews.
Requirements
- Experience working with users and engineering teams to identify AI opportunities and agree success criteria focused on outcomes, productivity, and quality.
- Hands-on experience improving AI-assisted engineering workflows using AI coding assistants or agent frameworks, including model interactions, orchestration, context management, and tool use.
- Experience creating representative evaluation datasets, controlled comparisons, repeatable measurements, and evidence-based trade-off analyses.
- Experience helping teams adopt and refine AI workflows through feedback, guidance, and documentation.
- Software engineering foundations and practical object-oriented programming experience in a language such as C#, Java, or C++, including testing, debugging, API integration, and automated quality checks.
- Strong communication, collaboration, peer review, and delivery skills.
- Practical C# development experience is preferred.
- Experience with rubric-based scoring, model-based judging, human-review calibration, workflow benchmarking, or integrating AI evaluations into CI/CD pipelines is preferred.
Benefits
- Arm will meet employer costs associated with skilled worker sponsorship when required, while the successful candidate is responsible for individual application fees and the Immigration Health Surcharge.
- Hybrid working patterns are determined by each team, with role-specific details shared during the application process.
- Recruitment adjustments and accommodations are available upon request.
Categories
About Arm
Arm’s foundational technology is defining the future of computing. A future built by the greatest technology ecosystem in the world. A future built on Arm. Arm is everywhere technology matters. Technology matters everywhere. Together, we’ll power every technology revolution moving forward, including cloud computing, automotive and autonomous systems, IoT, the metaverse, and beyond. Changing the world. Again. On Arm.
