4 months ago
Bengaluru, IndiaSenior
Responsibilities
- Develop automated testing frameworks for model inference, prompt pipelines, agentic workflows, RAG/retrieval systems, and serving APIs.
- Own end-to-end quality for ML models, AI agents, and AI-powered product features.
- Create evaluation datasets and ground-truth sets, measure response accuracy and consistency, and build LLM-as-judge and automated scoring pipelines.
- Build regression suites to detect quality drift caused by model, prompt, embedding, or training-data changes.
- Validate model correctness, accuracy, latency, behavioral consistency, product integration, end-to-end flows, and fallback behavior.
- Identify hallucinations, unsafe outputs, edge cases, and other generative AI failure modes.
- Use AI and LLMs to generate test cases, synthetic data, and automation scaffolding.
- Perform root cause analysis for production model and behavioral issues and partner with ML Engineers on preventive improvements.
- Define quality metrics, guardrails, and automated monitoring for model degradation.
- Promote quality and evaluation engineering best practices across the ML Engineering organization.
Requirements
- 5–8 years of experience in SDET, quality engineering, or software engineering with a strong record of building test automation frameworks.
- Strong programming skills in Python and/or Java with the ability to write production-quality automation code.
- Hands-on experience testing data-intensive or ML/AI systems, or a strong software-testing background with demonstrated ML/LLM fluency.
- Understanding of model training and inference, evaluation metrics such as precision, recall, and F1, and non-deterministic model outputs.
- Experience designing evaluation frameworks or working with evaluation datasets, benchmarks, or LLM-as-judge approaches.
- Familiarity with prompting, RAG, embeddings, vector search, hallucinations, drift, and prompt sensitivity.
- Experience with CI/CD, test orchestration, and API/integration testing.
- Strong analytical, debugging, root-cause-analysis, collaboration, and communication skills.
- Preferred experience with ML/evaluation tools including MLflow, Weights & Biases, LangSmith, Ragas, DeepEval, or provider evaluation suites.
- Preferred experience with AWS, GCP, or Azure; Docker and Kubernetes; model observability, drift detection, production ML monitoring, agentic systems, tool use, or multi-step LLM workflows.
- Experience in a high-scale SaaS or data platform environment is preferred.
Tech Stack
Categories
About Tekion
Tekion builds an AI-native, cloud platform for automotive retail that unifies dealers, OEMs, and partners. Its products include Automotive Retail Cloud (a dealership management system for retailers), Automotive Enterprise Cloud for manufacturers, and Automotive Partner Cloud for integrations, delivered as subscription software. Privately held and headquartered in Pleasanton, California, Tekion raised private equity funding in 2024.