6 days ago
Remote, United KingdomMid Level
Responsibilities
- Build agents and configure agentic workflows using system prompts, model selection, and tool integrations.
- Run end-to-end evaluations, create and curate datasets, analyze results, and report on agent quality and reliability.
- Define and execute QA processes for framework features, integrations, and releases, including identifying bugs, edge cases, regressions, and drift.
- Onboard, troubleshoot, and support technical teams adopting the agentic systems framework.
- Gather user feedback, identify friction points, and communicate improvement opportunities to engineering.
- Create documentation, guides, and examples for teams building with the framework.
- Conduct discovery, translate user needs into specifications, and contribute to prioritization and roadmap discussions.
- Run experiments, compare agent configurations, define quality criteria, and report on evaluation metrics.
Requirements
- At least 3 years of experience in a role combining development, QA, testing, or technical product work.
- Experience building or configuring AI/ML-powered systems or applications.
- Experience working with end users or internal teams to gather requirements and feedback.
- Comfort with APIs, configuration interfaces, and developer tooling.
- Understanding of LLM application integration, including prompting, tool calling, and structured outputs.
- Strong analytical and communication skills with attention to detail and a quality-first mindset.
- Experience with discovery, specification writing, prioritization, or other product management practices.
- Bachelor’s degree or higher in Computer Science, Engineering, or a related field, or equivalent practical experience.
- Excellent written and conversational English at C1 level or higher.
- Preferred: experience with agentic systems or LangGraph, LangChain, or similar frameworks.
- Preferred: experience with observability or evaluation tooling such as LangFuse, MCP Servers/Clients, QA engineering, test automation, or testing AI and non-deterministic systems.
- Preferred: experience in product management or technical product ownership, Python development, vector databases, RAG pipelines, NLP, and technical documentation.
Benefits
- Remote work environment with continuous learning and professional development opportunities.
- Competitive salary and benefits package.
- Supportive collaborative team culture.
- Occasional travel for training and industry events.
Tech Stack
Categories
