3 months ago
Bengaluru, IndiaSenior / Staff+
Responsibilities
- Develop a deep understanding of Tekion’s AI agents, ML models, business domains, and data.
- Own end-to-end quality for ML models, AI agents, and AI-powered features shipped by the ML Engineering team.
- Design and implement automated testing frameworks for model inference, prompt pipelines, agentic workflows, retrieval systems, and serving APIs.
- Validate model correctness, accuracy, latency, and behavioral consistency across model versions, prompt changes, and upgrades.
- Create and curate evaluation datasets and ground-truth sets, measure response quality, identify hallucinations and unsafe outputs, and build automated scoring pipelines.
- Use AI and LLMs to generate test cases, synthetic test data, and automation scaffolding.
- Build regression suites that detect quality drift when models, prompts, embeddings, or training data change.
- Validate ML model and agent integration into Automotive Retail Cloud products, including end-to-end flows and fallback behavior.
- Drive root-cause analysis for production model and behavioral issues and partner with ML Engineers on preventive improvements.
- Define quality metrics, guardrails, and automated monitoring for model degradation.
- Champion quality and evaluation engineering best practices across the ML Engineering organization.
Requirements
- 5–8 years of experience in SDET, quality engineering, or software engineering, with a strong record of building test automation frameworks.
- Strong programming skills in Python and/or Java, with the ability to write production-quality automation code.
- Hands-on experience testing data-intensive or ML/AI systems, or a strong software-testing background with demonstrated ML/LLM fluency.
- Understanding of model training and inference, evaluation metrics such as precision, recall, and F1, and the non-deterministic nature of model outputs.
- Experience designing evaluation frameworks or working with evaluation datasets, benchmarks, or LLM-as-judge approaches.
- Familiarity with prompting, retrieval-augmented generation, embeddings, vector search, hallucinations, drift, and prompt sensitivity.
- Experience with CI/CD, test orchestration, and API or integration testing.
- Strong analytical, debugging, and root-cause-analysis skills across model, data, and code layers.
- Excellent collaboration and communication skills for partnering with ML, data, and product teams.
- Preferred experience with ML and evaluation tooling such as MLflow, Weights & Biases, LangSmith, Ragas, DeepEval, or provider evaluation suites.
- Preferred experience with AWS, GCP, or Azure and containerized environments using Docker and Kubernetes.
- Preferred exposure to model observability, drift detection, production ML monitoring, agentic systems, tool use, or multi-step LLM workflows.
- Prior experience in a high-scale SaaS or data-platform environment is preferred.
Tech Stack
Categories
About Tekion
Tekion builds an AI-native, cloud platform for automotive retail that unifies dealers, OEMs, and partners. Its products include Automotive Retail Cloud (a dealership management system for retailers), Automotive Enterprise Cloud for manufacturers, and Automotive Partner Cloud for integrations, delivered as subscription software. Privately held and headquartered in Pleasanton, California, Tekion raised private equity funding in 2024.