6 days ago
Bengaluru, IndiaSenior
Responsibilities
- Build automated testing suites for hallucinations, bias, toxicity, prompt injection, RAG systems, multi-agent workflows, conversational AI, and prompt regressions.
- Statistically validate AI outputs and audit data ingestion, transformation, feature-store, vector database, embedding, and synthetic-data quality.
- Maintain automated suites for ML metrics and deep-learning loss curves and monitor data and concept drift on live inference endpoints.
- Build test automation frameworks for APIs, backend services, and model endpoints.
- Integrate AI evaluation and data quality suites into MLOps and CI/CD pipelines so quality failures block releases.
- Define AI quality KPIs and communicate release readiness to engineering and product teams.
Requirements
- 5+ years of SDET experience.
- Expert-level Python experience with test automation, evaluation pipelines, and data analysis using Pandas, NumPy, and Pytest.
- SQL experience for data-output validation, ground-truth querying, and pipeline quality checks.
- Experience with LLM evaluation frameworks such as RAGAS, TruLens, DeepEval, or Promptflow.
- Experience with LangChain, LangSmith, or LlamaIndex for agent workflow testing, prompt tracing, and response debugging.
- Experience testing OpenAI, Anthropic, or Hugging Face APIs and vector databases.
- Experience with Postman, REST Assured, or Requests for API contract and integration testing.
- Experience with MLflow, Docker, GitHub Actions, and Jenkins.
- Experience with Grafana, Kibana, or OpenTelemetry for monitoring and distributed tracing.
- Preferred: experience with AWS Bedrock, Azure OpenAI, GCP Vertex AI, Kubeflow, Weights & Biases, Feast, Scikit-learn, TensorFlow, PyTorch, Kubernetes, Terraform, Playwright, Cypress, Locust, or JMeter.
- Preferred: knowledge of statistical hypothesis testing, synthetic data generation, and AI/model evaluation datasets.
Benefits
- Opportunity to work with experienced teams on complex challenges at scale.
- Impactful role shaping and securing a global, cloud-native, AI-driven platform.
- Innovative, collaborative, and fast-paced work culture.
Tech Stack
Categories
About Tekion
Tekion builds an AI-native, cloud platform for automotive retail that unifies dealers, OEMs, and partners. Its products include Automotive Retail Cloud (a dealership management system for retailers), Automotive Enterprise Cloud for manufacturers, and Automotive Partner Cloud for integrations, delivered as subscription software. Privately held and headquartered in Pleasanton, California, Tekion raised private equity funding in 2024.