12 days ago
Base Salary
$388k - $619k/yr
Responsibilities
- Build observability frameworks that provide metrics, logs, and distributed traces across online inference, batch scoring, feature pipelines, and agent orchestration.
- Create reusable primitives for monitoring model performance, data quality, drift, degradation, latency, cost, and reliability.
- Develop evaluation frameworks for LLM and agentic systems covering response quality, grounding, hallucination, task success, tool use, trajectory correctness, LLM-as-a-judge, and human review.
- Lead build-versus-buy evaluations and own SDKs, connectors, and APIs for vendor platform integration.
- Build reusable libraries, SDKs, and templates that make observability and evaluation the default for new systems.
- Provide dashboarding, alerting, and SLO/SLI building blocks for ML and product teams.
- Partner with engineering, product, machine learning, and data teams to turn needs into reusable platform capabilities.
Requirements
- Experience in software, AI/ML, or platform engineering with production observability, monitoring, or ML/LLM evaluation experience.
- Track record designing standards, libraries, SDKs, or frameworks adopted by other teams.
- Strong Python skills and proficiency in at least one of Java, Go, or Scala, with experience building production services.
- Practical experience operating ML models in production through online serving and/or batch processing, including drift, model performance, and data quality.
- Hands-on experience with modern observability stacks such as Prometheus/Grafana, Datadog, OpenTelemetry, ELK/OpenSearch, or Jaeger/Tempo.
- Understanding of distributed systems, microservices, and at least one major cloud platform such as AWS, GCP, or Azure.
- Experience with vendor integration and VPC deployment.
- Ability to collaborate cross-functionally and communicate clearly about system behavior, quality, and risk.
- Experience using AI tools as a core part of the engineering workflow.
- Preferred experience with ML/LLM observability and evaluation tools such as Arize, Braintrust, LangFuse, Weights & Biases, Galileo, Vertex AI Model Monitoring, or SageMaker Model Monitor.
- Preferred experience building or shipping LLM/GenAI applications and evaluating prompts, results, metrics, LLM-as-a-judge systems, and human-in-the-loop review.
- Preferred experience evaluating agentic systems, including tool use, multi-step reasoning, and trajectory or task-success measurement.
Categories
About Netflix
Netflix builds and operates a global streaming service for TV series, films, games, and live programming, and produces original content through its in-house studio. It serves consumers in 190+ countries via subscription and ad-supported plans on internet-connected devices. Founded in 1997 and headquartered in Los Gatos, California, Netflix is a public company traded on NASDAQ (NFLX).