Machine Learning Engineer II, GenAI Evaluation Framework
Cox AutomotiveResponsibilities
- Own the GenAI Evaluation Framework roadmap, prioritization, technical direction, and deprecation decisions.
- Build and maintain LLM-as-judge, rule-based, and statistical evaluators calibrated against human-labeled baselines.
- Develop and operate evaluation pipelines that score production agent traffic and publish results to dashboards and the Cox Agent Registry.
- Assemble, version, and maintain golden datasets with product teams and subject matter experts.
- Integrate the framework into adopting teams’ agents, debug failures, and create documentation and reference implementations.
- Publish framework adoption, agent quality health, and cost-per-quality metrics with audit trails for governance reviews.
- Add tracing and logging to agent workflows and integrate evaluation checks into continuous integration and deployment pipelines.
- Onboard three to four product teams and establish reliable monitored pipelines with operational runbooks.
Requirements
- Two or more years of professional machine learning engineering experience with a bachelor’s degree, or up to two years of combined experience with a master’s degree in a quantitative field.
- Current authorization to work in the United States for any employer without current or future sponsorship; OPT, CPT, STEM/OPT, and visa sponsorship are not accepted.
- Current residence within commuting distance of Atlanta.
- Experience owning the direction of a technical product, internal tool, library, or platform used by other teams.
- Ability to maintain roadmaps, prioritize competing requests, and explain tradeoffs to engineering and product stakeholders.
- Strong Python skills and working experience with testing, version control, code review, and continuous integration.
- Hands-on experience with large language models, including prompting, API integration, retrieval, or agent frameworks.
- Working knowledge of measurement concepts such as sampling, inter-rater agreement, precision and recall, or comparative testing.
- Experience with cloud services, ideally AWS, and with building data pipelines.
- Strong written communication for documentation, roadmap updates, and senior-stakeholder reporting.
- Preferred qualifications include formal technical product ownership, experience with developer tools, infrastructure, or ML platforms, and experience in federated organizations where adoption is voluntary.
- Preferred experience includes DeepEval, AWS Bedrock AgentCore Evaluations, Langfuse, RAGAS, OpenTelemetry, Datadog, model risk, AI governance, regulated environments, and collaboration between data science and product or platform engineering teams.
Categories
About Cox Automotive
Cox Automotive is the world’s largest automotive services and technology provider. Fueled by the largest breadth of first- and third-party data fed by 2.3 billion online interactions a year, Cox Automotive tailors leading solutions for car shoppers, auto manufacturers, dealers, lenders and fleets. The company has 29,000+ employees on four continents and a portfolio of industry-leading brands that include Autotrader®, Kelley Blue Book®, Manheim®, vAuto®, Dealertrack®, NextGear Capital™, CentralDispatch® and Cox Fleet®. Cox Automotive is a subsidiary of Cox Enterprises Inc., a privately owned, Atlanta-based company with $23 billion in annual revenue. Cox Automotive has been included on Glassdoor’s Best Companies in Tech & AI 2026 and Best Place to Work in 2026 lists.