about 5 hours ago
Responsibilities
- Build automated testing suites for LLM-powered products to detect hallucinations and vulnerabilities.
- Implement evaluations for RAG systems measuring context relevance and answer faithfulness.
- Design test beds to validate multi-agent workflows and tool-calling accuracy.
- Create automated conversation simulations to stress-test agent behavior.
- Statistically validate AI data outputs to catch data quality failures.
- Audit data ingestion and transformation pipelines for schema drift.
- Maintain automated suites tracking ML metrics and deep learning loss curves.
- Build and maintain scalable test automation frameworks for APIs and backend services.
- Define and track AI quality KPIs and communicate release readiness.
Requirements
- 8+ years of experience in a Software Development Engineer in Test (SDET) role.
- Expert level proficiency in Python for test automation and data analysis.
- Strong SQL skills for data output validation and quality checks.
- Experience with LLM evaluation frameworks like RAGAS and DeepEval.
- Familiarity with MLOps tools such as MLflow and CI/CD practices.
- Knowledge of observability tools like Grafana and Kibana.