AI Evaluation Engineer
Judi Health4 days ago
Charlotte, NC, USA +2 moreMid Level
Base Salary
$135k - $200k/yr
Responsibilities
- Build data evaluation pipelines for production conversations and agent interactions, including session reconstruction from traces, logs, recordings, and transcripts.
- Apply human-feedback signals and automated evaluators such as LLM-as-judge systems to label and score evaluations.
- Own recurring automated evaluation runs against staging and production and define benchmarks for accuracy, reliability, and safety.
- Design and extend a unified evaluation framework for multiple agent types and workflows.
- Translate product expectations into measurable success criteria and metrics.
- Build APIs and internal tools for scenario curation, dataset management, and evaluation execution.
- Manage dataset versioning, provenance, and lifecycle.
- Develop dashboards, monitoring, and alerting for AI quality and safety signals.
- Integrate evaluations into CI/CD pipelines for scheduled and gated runs.
- Provide experiment tracking, reproducibility, and comparison across prompt, model, and agent experiments.
- Partner with data science, engineering, and product teams to define and improve AI behavior in production.
Requirements
- At least 4 years of experience in data engineering, ML engineering, or software engineering.
- Bachelor’s or Master’s degree in Computer Science, Machine Learning, or a related quantitative field.
- Strong proficiency in Python and SQL.
- Experience building and maintaining production data pipelines.
- Experience with at least one cloud platform, with AWS preferred.
- Prior work on LLM or agent evaluation infrastructure is a plus.
- Familiarity with metrics for safety, reliability, or quality in AI systems is a plus.
- Experience with voice or call-center data, including audio, transcripts, or sentiment, is a plus.
- Experience with browser automation tools such as Playwright for end-to-end evaluations is a plus.
- Deep SQL expertise is a plus.
Benefits
- Hybrid work arrangement requiring three days in the office, with offices in New York City, Denver, and the Charlotte area.