about 5 hours ago
Responsibilities
- Build automated testing suites for LLM-powered products to detect hallucinations and vulnerabilities.
- Implement automated evaluations for RAG systems measuring context relevance and answer faithfulness.
- Design test beds to validate multi-agent workflows and tool-calling accuracy.
- Create prompt regression frameworks to assess output consistency based on system changes.
- Statistically validate AI data outputs to catch silent data quality failures.
- Audit data ingestion and transformation pipelines for schema drift and data corruption.
- Maintain automated suites tracking ML metrics and deep learning loss curves.
- Build and maintain scalable test automation frameworks for APIs and backend services.
- Define and track AI quality KPIs and communicate release readiness to teams.
Requirements
- 5+ years of experience in a Software Development Test Engineer role.
- Expert level proficiency in Python for test automation and data analysis.
- Strong SQL skills for data output validation and quality checks.
- Experience with LLM evaluation frameworks like RAGAS and DeepEval.
- Familiarity with MLOps tools such as MLflow and CI/CD practices.
- Knowledge of statistical analysis techniques for output validation.