4 months ago
Bengaluru, IndiaStaff+
Responsibilities
- Own the end-to-end QA strategy and quality roadmap for AI copilots, agent workflows, enterprise deployments, and platform features.
- Lead, mentor, and scale QA engineers and automation specialists while partnering with Product, ML Engineering, and Platform Engineering.
- Define quality gates, release-readiness standards, and the release sign-off process.
- Develop LLM and AI-agent evaluation systems covering accuracy, hallucination detection, factuality, safety, bias, reasoning, tool use, workflows, and regression scenarios.
- Architect automation frameworks and automated API, microservices, backend, and cloud-system testing.
- Integrate automated test suites into CI/CD pipelines and manage load, stress, spike, endurance, and soak testing.
- Prepare QA strategies, test plans, evaluation reports, execution metrics, risk communications, and impact analyses.
- Drive root-cause analysis, issue dashboards, continuous QA improvement, and monitoring of model drift and performance anomalies.
Requirements
- 12+ years of experience in Software QA.
- Strong hands-on experience with test automation frameworks and scripting in Python, Java, or JavaScript.
- Experience with microservice-based architectures, distributed cloud systems, APIs, and high-performance backend services.
- Working knowledge of AI/ML systems, datasets, model evaluation, vector stores, RAG, and LLM operations.
- Experience architecting and managing automation frameworks using tools such as Playwright, Selenium, PyTest, and Postman.
- Experience with cloud architecture running on Azure, AWS, or GCP.
- Strong analytical and problem-solving skills with the ability to drive organization-wide quality initiatives.
- Preferred: experience testing LLMs, RAG pipelines, or autonomous AI agents; familiarity with AI observability, logging, and monitoring stacks; exposure to security testing, performance/load testing, SOC2, and GDPR; and experience in a fast-paced zero-to-one startup environment.
Benefits
- Opportunity to build foundational, practical, auditable, and human-aligned AI systems for enterprise use.
- Work with deep technology, real-world enterprise workflows, and systems designed for production scale.
- Headquartered in Los Altos, California.
Tech Stack
Apache JMeterAWSAzureDatadogGitHub ActionsGoogle Cloud PlatformJavaJavaScriptJenkinsPostmanPrometheuspytestPythonSelenium
Categories
