Sr Product Software Engineer I
The Walt Disney Company20 hours ago
Bengaluru, IndiaSenior
Responsibilities
- Implement automated benchmarking workflows for model versions and providers.
- Build regression detection systems that compare model outputs across updates.
- Develop evaluation harnesses for structured test datasets.
- Integrate evaluation results into CI/CD release workflows and automated release gating.
- Support A/B testing, response consistency analysis, hallucination analysis, and prompt or workflow drift detection.
- Ensure evaluation results are reproducible and traceable.
- Partner with AI Core Engineering to validate model integrations and with Runtime & Reliability teams to correlate evaluation results with production performance.
- Participate in model readiness reviews and certification processes.
Requirements
- Bachelor’s degree in Computer Science, Engineering, or a related technical field, or equivalent experience; a master’s degree is preferred.
- 5+ years of backend or systems engineering experience.
- Strong proficiency in Python.
- Experience building automated testing or evaluation pipelines.
- Experience integrating external APIs or model services.
- Experience working in cloud-based environments.
- Preferred experience includes AI-powered systems or LLM integrations, benchmarking, structured dataset validation, experimentation or validation platforms, distributed systems, observability or analytics systems, and cloud-native deployments.
About The Walt Disney Company
The Walt Disney Company is a diversified entertainment business that produces films and television, operates theme parks and resorts, and runs streaming services including Disney+, Hulu, and ESPN+. It earns revenue from subscriptions and advertising, theatrical distribution, licensing and merchandise, and park admissions and experiences. Founded in 1923 and headquartered in Burbank, California, Disney owns brands such as Pixar, Marvel, Lucasfilm, and 20th Century Studios and serves audiences worldwide.