3 hours ago
Bengaluru, IndiaMid Level
Responsibilities
- Build and own the end-to-end QA strategy for AI copilots, workflows, and enterprise deployments.
- Lead, mentor, and scale a team of QA engineers and automation specialists.
- Partner with Product, ML Engineering, and Platform Engineering to embed quality into early design phases.
- Define quality gates and release-readiness standards and own release sign-off for AI software and platform features.
- Develop evaluation systems for LLM accuracy, hallucination detection, factuality, safety, bias mitigation, agent reasoning, tool use, and multi-step workflows.
- Create synthetic datasets and scenario-based tests for edge cases and regression testing.
- Validate model performance across fine-tuning, RAG, knowledge-grounding, and inference scenarios.
- Architect and manage automation frameworks and integrate automated suites into CI/CD pipelines.
- Automate API, microservices, backend, and cloud-system testing.
- Conduct load, stress, spike, endurance, and soak testing.
- Perform root-cause analysis, maintain issue dashboards, and drive continuous QA improvement.
- Collaborate with DevOps and MLOps teams to monitor model drift, performance anomalies, and post-production behavior.
- Prepare QA strategies, test plans, evaluation reports, execution metrics, and leadership communications.
Requirements
- 3+ years of experience in software QA.
- Strong hands-on experience with test automation frameworks and scripting in Python, Java, or JavaScript.
- Experience working with microservice-based architectures, distributed cloud systems, APIs, and high-performance backend services.
- Working knowledge of AI/ML systems, datasets, model evaluation, vector stores, RAG, and LLM operations.
- Experience architecting and managing automation frameworks using tools such as Playwright, Selenium, PyTest, and Postman.
- Experience with automated API, microservices, and backend testing on cloud architectures using Azure or AWS.
- Strong analytical and problem-solving skills with the ability to drive organization-wide quality initiatives.
- Preferred experience testing LLMs, RAG pipelines, or autonomous AI agents.
- Preferred familiarity with AI observability, logging, and monitoring stacks such as ELK, Prometheus, or Datadog.
- Preferred exposure to security testing, performance/load testing, and compliance requirements including SOC 2 and GDPR.
- Ability to work in a fast-paced, zero-to-one startup environment.
Tech Stack
Apache JMeterAWSAzureDatadogGitHub ActionsGoogle Cloud PlatformJavaJavaScriptJenkinsPostmanPrometheuspytestPythonSelenium
Categories
About JazzX AI
JazzX AI builds AI-native digital workers that execute governed, explainable workflows for enterprises, starting with lending and due-diligence processes. The company offers enterprise software and AI infrastructure to capture expert judgment, automate decisions, and run complex, regulated operations at scale. Founded in 2024 and privately held, it is based in California and backed by SAI Group.