2 months ago
Bengaluru, IndiaStaff+
Responsibilities
- Build automated suites for detecting hallucinations, bias, toxicity, prompt injection, RAG quality issues, and multi-agent workflow failures.
- Create conversation simulations and prompt regression frameworks to test agent behavior, output consistency, and multi-turn interactions.
- Statistically validate AI outputs and audit data ingestion, transformation, feature-store, vector database, embedding, and synthetic-data quality.
- Maintain automated tracking for ML metrics, deep-learning loss curves, data drift, and concept drift across model versions and live endpoints.
- Build test automation for APIs, backend services, and model endpoints, and integrate quality suites into MLOps and CI/CD pipelines.
- Define AI quality KPIs and communicate release readiness to engineering and product teams.
Requirements
- 8+ years of experience in an SDET role.
- Expert-level Python experience for test automation, evaluation pipelines, and data analysis.
- SQL experience for data output validation, ground-truth querying, and pipeline quality checks.
- Experience with LLM evaluation, agent workflow testing, prompt tracing, API automation, MLOps, CI/CD, and observability tools.
- Preferred experience with AWS Bedrock, Azure OpenAI, GCP Vertex AI, Kubeflow, Weights & Biases, Feast, Scikit-learn, TensorFlow, PyTorch, Kubernetes, Terraform, Playwright, Cypress, Locust, or JMeter.
Tech Stack
Categories
About Tekion
Tekion builds an AI-native, cloud platform for automotive retail that unifies dealers, OEMs, and partners. Its products include Automotive Retail Cloud (a dealership management system for retailers), Automotive Enterprise Cloud for manufacturers, and Automotive Partner Cloud for integrations, delivered as subscription software. Privately held and headquartered in Pleasanton, California, Tekion raised private equity funding in 2024.