Machine Learning Engineer II, GenAI Evaluation Framework
Cox Enterprises, Inc.1 day ago
Atlanta, GA, USAMid Level
Base Salary
$92k - $154k/yr
Responsibilities
- Own the GenAI Evaluation Framework roadmap, prioritization, technical direction, and deprecation decisions.
- Sequence and negotiate onboarding for product teams adopting the framework.
- Build and maintain LLM-as-judge, rule-based, and statistical evaluators calibrated against human-labeled baselines.
- Build and operate scheduled evaluation pipelines that publish results to dashboards and the Cox Agent Registry.
- Assemble, version, and maintain golden datasets with product teams and subject matter experts.
- Integrate the framework into adopting teams’ agents, debug failures, and create documentation and reference implementations.
- Publish monthly framework adoption, agent quality health, and cost-per-quality metrics with governance audit evidence.
- Add tracing and logging to agent workflows and integrate evaluation checks into continuous integration and deployment pipelines.
- Establish reliable monitored pipelines, runbooks, and self-service adoption capabilities for multiple product teams.
Requirements
- At least two years of professional machine learning engineering experience with a bachelor’s degree, or up to two years of combined experience with a master’s degree in a quantitative field.
- Currently authorized to work in the United States for any employer without current or future sponsorship; OPT, CPT, STEM/OPT, and visa sponsorship are not accepted.
- Currently living within a commutable distance to Atlanta.
- Experience owning the direction of a technical product, internal tool, library, or platform used by other teams.
- Ability to maintain roadmaps, prioritize competing requests, and explain tradeoffs to engineering and product stakeholders.
- Strong Python skills and working practices involving testing, version control, code review, and continuous integration.
- Hands-on experience with large language models, including prompting, API integration, retrieval, or agent frameworks.
- Working knowledge of sampling, inter-rater agreement, precision and recall, or comparative testing.
- Experience with cloud services, ideally AWS, and building data pipelines.
- Strong written communication skills for documentation, roadmap updates, and senior-stakeholder reporting.
- Preferred experience with developer tools, infrastructure, or ML platform product ownership; evaluation tools such as DeepEval, AWS Bedrock AgentCore Evaluations, Langfuse, or RAGAS; federated adoption models; observability and tracing tools; AI governance or regulated environments; and collaboration between data science and product or platform engineering.
Benefits
- Hybrid work arrangement with the ability to work remotely part of the week.
- The role is based within commuting distance of Atlanta and involves approximately 5% travel.
- Eligible employees may take flexible paid vacation consistent with their duties and company needs.
- Seven paid holidays and up to 160 hours of paid annual wellness time are offered.
- Additional leave may include bereavement, voting, jury duty, volunteer, military, and parental leave.
- Benefits may include medical, dental, and vision insurance, retirement planning through a 401(k), sick leave, parental leave, flexible vacation, wellness days, and/or PTO.
- Employment requires passing a pre-employment drug test; marijuana is not currently included in testing for this position.
Categories
About Cox Enterprises, Inc.
Cox Enterprises is a privately held, Atlanta-based company operating businesses in broadband and automotive services. Through Cox Communications, it provides internet, voice, and video to residential and business customers; through Cox Automotive, it runs marketplaces and dealer software including Autotrader, Kelley Blue Book, and Manheim. Founded in 1898, the family-owned company also invests in sustainable technologies and other growth ventures.