4 months ago
Bengaluru, IndiaStaff+
Responsibilities
- Build and own Tekion’s shared AI evaluation infrastructure and frameworks.
- Create and maintain evaluation datasets, golden sets, automated scoring pipelines, and quality metrics.
- Implement LLM-as-judge, rubric-based, reference-based, offline, and online evaluation methods.
- Identify hallucinations, unsafe or biased outputs, edge cases, drift, and other generative failure modes.
- Establish evaluation gates in CI/CD and monitor model, prompt, and data changes before and after release.
- Develop dashboards and reporting that make AI quality actionable for ML and product teams.
- Use LLMs for automated judging and synthetic data generation while championing responsible-AI quality practices.
Requirements
- 8+ years of experience in SDET, quality engineering, ML engineering, or data science, with hands-on experience building evaluation or measurement systems; strong SDET candidates with deep LLM/ML fluency are also considered.
- Strong Python programming skills for building reusable evaluation pipelines and tooling.
- Deep understanding of ML/LLM evaluation, benchmark design, and evaluating non-deterministic systems.
- Hands-on experience with LLM-as-judge, rubric-based scoring, or human-in-the-loop evaluation.
- Understanding of prompting, RAG, embeddings, tool use, hallucination, drift, prompt sensitivity, and bias.
- Experience designing and curating datasets, including labeling or annotation strategy and data quality.
- Strong statistical intuition and ability to communicate quality signals to ML and product teams.
- Preferred experience with Ragas, DeepEval, LangSmith, TruLens, Promptfoo, HELM, provider evaluation suites, online evaluation, guardrails, production model monitoring, responsible AI, red-teaming, MLflow, or Weights & Biases.
About Tekion
Tekion builds an AI-native, cloud platform for automotive retail that unifies dealers, OEMs, and partners. Its products include Automotive Retail Cloud (a dealership management system for retailers), Automotive Enterprise Cloud for manufacturers, and Automotive Partner Cloud for integrations, delivered as subscription software. Privately held and headquartered in Pleasanton, California, Tekion raised private equity funding in 2024.