AI QA Engineer

2 days ago

Noida, IndiaMid Level

Responsibilities

Own the test strategy for LLM features, agentic workflows, RAG pipelines, and ML models.
Write and execute test plans, test cases, test reports, exploratory tests, and defect-lifecycle activities.
Define acceptance criteria and quality metrics for accuracy, groundedness, hallucination rate, relevance, cost, and other non-deterministic outputs.
Build golden datasets, ground-truth labels, and regression suites with subject-matter experts.
Develop automated AI evaluation pipelines using DeepEval, Ragas, promptfoo, and LangSmith.
Test agent planning, tool calling, integrations, error handling, fallbacks, and guardrails.
Validate ML models and data pipelines for data quality, segment performance, bias, fairness, and drift.
Perform adversarial and safety testing for prompt injection, jailbreaks, PII leakage, and compliance.
Monitor production AI quality, triage degradations, lead root-cause analysis, and support release sign-off.
Partner with Data Science, AI, engineering, and business stakeholders across time zones.

Bachelor’s or master’s degree in Computer Science, Data Science, ML/AI, Statistics, or a related field.
2–3 years of experience in AI/ML, data science, or AI-focused QA, including at least 1 year of hands-on test automation.
Working knowledge of ML model building, evaluation metrics, LLM behavior, prompting, context, and RAG.
Experience evaluating GenAI applications, RAG pipelines, agentic workflows, and function calling.
Strong Python skills with PyTest, API testing, and CI/CD integration.
Knowledge of statistics and experimentation, including sampling, significance testing, and model A/B comparison.
Experience with SQL and data validation tools such as Great Expectations or Evidently.
Experience testing document and image analysis solutions using unstructured data.
Ability to define pass/fail criteria and rubrics for non-deterministic outputs and translate business requirements into measurable quality standards.
Clear written and verbal communication with technical and business stakeholders.
Preferred: synthetic and privacy-safe test data generation, Docker, Kubernetes, MLOps tooling, AI governance frameworks, LangChain, LangGraph, Hugging Face, Anthropic, OpenAI, open-source models, Playwright, and Selenium.

Competitive benefits for the countries served by apexanalytix.
BeWell@apex wellness initiative covering emotional, physical, community, financial, social, and intelligence wellness.
Mentor Program and Internal Training Portal.
Education, tuition, and certification assistance.
Professional growth and development resources.