Lead AI QA Engineer
Kobie Marketing5 months ago
Bengaluru, IndiaStaff+
Responsibilities
- Design and build Python evaluation harnesses with golden datasets, LLM-as-judge graders, regression suites, and trace-based assertions.
- Author automated tests for prompts, tools, structured outputs, retrieval pipelines, and end-to-end agent workflows.
- Validate tool-execution guardrails, authentication scoping, PII protections, prompt-injection defenses, and hallucination mitigation.
- Integrate evaluations into CI using Dataiku Evaluations, GitHub Actions, or Jenkins against quality, safety, and cost targets.
- Instrument traces, monitor production drift, and use observability data to improve evaluation coverage.
- Define release criteria, run pre-production and shadow tests, and partner with engineering to resolve regressions.
- Develop Snowflake-backed retrieval testing patterns and collaborate with platform teams on observability, security, and cost.
- Establish AI and data engineering QA standards and share knowledge across India and U.S. teams.
Requirements
- At least 3 years of professional QA or SDET experience automating tests for backend services or data pipelines.
- At least 1 year of hands-on production experience testing LLM or AI features, including prompt regression, tool or function-call validation, structured outputs, and RAG correctness.
- Working knowledge of RAGAS, DeepEval, LangSmith, Langfuse, or comparable LLM-as-judge evaluation tooling.
- Strong Python and PyTest skills, solid SQL skills, and experience with at least one of AWS, Azure, or GCP.
- Experience with Git, Docker, REST APIs, and at least one CI tool such as GitHub Actions, Jenkins, GitLab CI, or CircleCI.
- Understanding of data security and responsible AI practices, especially in PCI-compliant or regulated environments.
- Ability to manage concurrent projects across teams and time zones and communicate with technical and non-technical stakeholders.
- A bachelor’s degree is not required; equivalent practical experience, including bootcamps, self-taught work, career changes, or non-CS technical degrees, is accepted.
- Preferred experience includes Dataiku DSS or Dataiku Evaluations, Dataiku LLM Mesh, Knowledge Banks, Prompt Studio, Visual or Code Agents, Snowflake, Snowpark, Snowflake Cortex, LLM red-teaming, prompt-injection testing, adversarial test generation, multi-agent patterns, Locust, JMeter, k6, and QA or AI testing certifications.
- Experience in loyalty, martech, adtech, or another data-rich B2B domain is preferred.
Benefits
- Competitive benefits, comprehensive health coverage, and well-being perks for employees and dependents.
- Public holidays, flexible time off, and a focus on work-life balance.
- Fully integrated collaboration with U.S. teams and exposure to global projects and industry leaders.
- A physical office in Bengaluru, India is planned as the Tech Hub grows.
- Collaborative, growth-focused culture with diversity, equity, inclusion, charitable partnerships, and volunteer opportunities.
Tech Stack
Categories
About Kobie Marketing
Kobie Marketing builds loyalty program technology and services for large consumer brands across retail, travel, financial services, telecom, and more. Its offering combines strategy consulting, analytics, and the Kobie Alchemy Loyalty Cloud platform to design, operate, and measure customer rewards and engagement programs. Founded in 1990 and headquartered in St. Petersburg, Florida, the privately held company supports enterprise clients with remote-first teams in the U.S. and India.