1 day ago
Remote, United StatesMid Level
Responsibilities
- Establish standardized automated evaluation frameworks for LLMs and traditional ML models using statistical validation, LLM-as-a-judge methods, ROUGE, METEOR, and related metrics.
- Define evaluation methodologies and quality criteria for LLM, RAG, and agent-based applications, including faithfulness, groundedness, relevance, task completion, tool-use correctness, and safety.
- Build reusable evaluation libraries, automated pipelines, regression tests, CI/CD quality gates, benchmark datasets, golden datasets, adversarial suites, and synthetic test data workflows.
- Implement offline and online evaluation, including production monitoring, sampling, drift detection, quality alerts, A/B testing, and shadow testing.
- Design, calibrate, and maintain LLM-as-a-judge evaluators and human annotation workflows, including rubric design, bias mitigation, and inter-annotator agreement practices.
- Deploy observability for model drift, hallucination risk, latency, throughput, token consumption, quality, and infrastructure cost.
- Conduct red-teaming and testing for hallucinations, toxicity, PII leakage, prompt injection, and jailbreaks.
- Produce evaluation evidence and reporting for security, privacy, compliance, responsible AI, and model risk reviews.
- Partner with product owners, engineers, data scientists, SMEs, and risk stakeholders to translate requirements into measurable thresholds.
- Coach product pods on evaluation practices, develop dashboards and reporting, and recommend evaluation tooling, platforms, standards, and best practices.
Requirements
- Master’s degree in computer science or a quantitative discipline is preferred.
- At least 4 years of experience in software, ML, or data engineering is required.
- At least 1–2 years of hands-on experience with LLM-based applications is required.
- Strong Python skills and software engineering fundamentals, including testing, version control, code review, and CI/CD, are required.
- Experience evaluating LLM/RAG systems, including retrieval metrics such as recall, precision, MRR, and nDCG, and generation metrics is required.
- Hands-on experience designing LLM-as-a-judge systems, prompt engineering, and judge validation is required.
- Working knowledge of statistics and experimental design, including sampling, confidence intervals, significance testing, and inter-rater agreement, is required.
- Experience with an evaluation or observability framework such as Ragas, DeepEval, promptfoo, LangSmith, Langfuse, MLflow, or an equivalent is required.
- Experience with cloud platforms such as Azure, AWS, or GCP and managed LLM services is required.
- Preferred experience includes evaluating agentic systems, ML evaluation, search/relevance, NLP, QA automation, LLMOps, monitoring, observability, production feedback loops, data annotation, and regulated-environment security practices.
- Familiarity with AI risk and governance frameworks such as NIST AI RMF or ISO/IEC 42001 and enterprise controls such as SSO, RBAC, private networking, logging, and encryption is preferred.
- Strong oral, written, presentation, communication, collaboration, troubleshooting, analytical, and prioritization skills are required.
Benefits
- Fully remote role open to U.S.-based candidates.
- Travel may be required.
- Reasonable accommodations will be considered when applicable and appropriate.
- The employer offers a caring culture, work-life balance, equal employment opportunity, and a drug-free workplace.
Tech Stack
Categories
About Sedgwick
Sedgwick provides third‑party claims administration, risk and benefits services for employers, insurers, and public entities, spanning workers’ compensation, property and liability, disability and absence, and product recall. It sells outsourced claims management and related technology on a fee‑for‑service basis, supported by analytics and AI‑enabled tools, across operations in more than 80 countries. Founded in 1969 and headquartered in Memphis, Tennessee, Sedgwick is privately held under The Carlyle Group.