about 4 hours ago
Bengaluru, IndiaSenior / Staff+
H1B Sponsor
Responsibilities
- Develop a deep understanding of Tekion’s AI agents and ML models.
- Design and own Tekion’s AI evaluation infrastructure as a shared capability.
- Create and maintain evaluation datasets and ground-truth sets.
- Define quality metrics for AI outputs including accuracy and consistency.
- Build automated scoring pipelines for evaluation methods.
- Validate user intents and measure response accuracy for AI capabilities.
- Identify hallucinations and design targeted evaluation suites.
- Establish evaluation gates in CI/CD for quality checks.
- Develop dashboards to make AI quality visible for teams.
- Use AI/LLMs to scale evaluation through automated judges.
Requirements
- 5–8 years in SDET, quality engineering, ML engineering, or data science.
- Strong programming skills in Python for building evaluation pipelines.
- Deep understanding of ML/LLM evaluation and benchmark design.
- Hands-on experience with LLM-as-judge and human-in-the-loop evaluation.
- Solid grasp of LLM/agent concepts and generative failure modes.
- Experience designing and curating datasets with data quality strategies.
- Strong statistical intuition for interpreting evaluation results.
- Excellent communication skills for translating quality signals.