AI Observability Engineer

4 months ago

Atlanta, GA, USAMid Level

Responsibilities

Monitor ML, LLM, and agentic AI performance using KPIs including context relevance, accuracy, precision, recall, and F1 scores.
Detect anomalies, bias, and data drift affecting the fairness and accuracy of AI systems.
Operate and evolve AI model management platforms, pipelines, KPIs, and model observability capabilities.
Review AI solutions for alignment with organizational policies, AI governance frameworks, and Responsible AI principles.
Support AI product teams through independent reviews, workshops, and responses to ad hoc queries.
Collaborate with legal, compliance, security, and other teams on Responsible AI issues.
Create executive-level Responsible AI reports and strategic recommendations.
Develop Responsible AI training content, tutorials, methods, tools, and other assets.
Monitor emerging global AI regulations, standards, and policies and translate them into framework updates.
Build or support production LLM, RAG, or model inference monitoring pipelines, dashboards, and alerts.

A bachelor’s degree and a minimum of three years of experience are required.
Knowledge of machine learning, large language models, algorithms, and evaluation metrics.
Ability to handle, clean, transform, and analyze data for monitoring purposes.
Familiarity with LLM development and the challenge of scaling human oversight of AI systems.
Experience or strong interest in Responsible AI or AI ethics and understanding of generative AI technologies and frameworks.
Ability to collaborate cross-functionally and implement Responsible AI practices, preferably in financial services or another regulated industry.
Working knowledge of policies and processes governing emerging AI tools and products.
Proven ability to solve complex problems, develop mitigation strategies, and influence stakeholders.
Preferred experience building production LLM, RAG, or model inference monitoring pipelines.
Preferred experience with OpenTelemetry distributed tracing or structured logs and AI observability tools such as Fiddler, Arize, or LangSmith.
Ability to build dashboards and alerts focused on AI model behavior beyond system uptime.

Primarily in-office work to support in-person collaboration.
Comprehensive benefits covering health, wellbeing, financial security, and professional development.
Continuous learning and development opportunities.
Inclusive culture supported by cultural awareness activities and Employee Resource Groups.
Reasonable accommodations are available for applicants and employees with disabilities.

10,000+ employees