11 hours ago
Hyderābād, IndiaSenior
Responsibilities
- Design and implement testing strategies for generative AI, LLM, RAG, agentic, and multi-agent systems.
- Build and maintain full-stack automated testing for Python backend services and TypeScript, React, and Angular frontends.
- Create integration, performance, UI, API, and specialized AI behavior tests for continuous evaluation.
- Develop and maintain AI quality frameworks, test documentation, test cases, and testing guidelines.
- Evaluate model responses, hallucinations, contextual accuracy, fairness, bias, edge cases, tool-use accuracy, and autonomous agent behavior.
- Test AI-specific security concerns including prompt injection and data leakage, and conduct performance, load, response-time, and resource-utilization testing.
- Curate and version golden datasets, implement evaluation harnesses, and use production tracing and feedback loops to identify drift and improve systems.
- Integrate automated quality suites into GitHub Actions and CI/CD pipelines and collaborate with Agile development teams in a regulated environment.
Requirements
- Bachelor’s or master’s degree in Computer Science, Software Engineering, or an equivalent degree.
- 7+ years of software testing experience, including at least 1 year focused on generative AI and/or agentic applications.
- Advanced Python proficiency and experience with testing frameworks such as pytest and Allure.
- Strong TypeScript test automation experience with tools such as Playwright, Jest, and Cypress.
- Expertise in AI evaluation frameworks such as RAGAS, DeepEval, and Promptfoo, including evaluation of LLM performance, hallucinations, and answer relevance.
- Experience testing autonomous agentic loops, tool use, reasoning verification, and goal-directed performance.
- Experience curating, managing, and versioning golden datasets for model benchmarking.
- Experience with tracing and observability tools such as LangSmith and Arize Phoenix.
- Understanding of LLM architectures, RAG systems, agents, AI failure modes, security testing, and AI quality practices.
- Experience working with cloud platforms such as AWS and with monitoring tools such as Grafana; familiarity with Jira and GitHub.
- Understanding of regulated environments, GxP and non-GxP products, CSV, ITSM, and Agile development practices is preferred.
- Strong communication, documentation, analytical, and problem-solving skills; English proficiency at C1+ level.
Benefits
- The role is based in Hyderabad, India, with working hours structured to overlap with Central European Time, typically extending into the IST evening.
- Employees work in an inclusive Roche culture that values personal expression, open dialogue, equal opportunity, and diverse perspectives.
Tech Stack
Categories
About Roche
Roche is a Swiss healthcare company that develops and sells prescription pharmaceuticals and in vitro diagnostic systems for hospitals, laboratories, and clinicians. Its business spans two core divisions—Pharmaceuticals and Diagnostics—with strengths in oncology, immunology, neuroscience, infectious diseases, and diabetes care. Founded in 1896 and headquartered in Basel, Switzerland, Roche also advances personalized healthcare by pairing therapies with companion diagnostics.