AI Quality & Evaluation Engineer, AI product
Woven by Toyota3 months ago
Tokyo, JapanMid Level
Responsibilities
- Define quality dimensions for AI products, including accuracy, consistency, safety, fairness, and UX.
- Conduct scenario-based testing, exploratory testing, and red teaming.
- Analyze LLM outputs to identify behavioral trends and failure patterns.
- Design evaluation processes using user feedback, logs, ratings, and inquiries.
- Standardize and automate reusable, scalable quality evaluation workflows.
- Collect and aggregate evaluation data and establish recurring quality checks.
- Build and operate quality feedback loops with development, MLOps, and data engineering teams.
- Document quality issues and risks and communicate them to relevant teams.
Requirements
- At least 3 years of QA or testing experience with products that use LLMs.
- Experience designing tests, including defining test perspectives and creating test cases.
- Ability to establish clear judgment criteria for systems with ambiguous specifications or non-unique correct answers.
- Experience automating test or evaluation processes.
- Experience collaborating with development teams to drive quality improvements.
- Business-level proficiency in both English and Japanese.
- Experience with scenario testing, red teaming, user-feedback and log analysis, and Japanese-language natural language processing is preferred.
Benefits
- Hybrid work arrangement with flexible working hours.
- Business trips to Woven City several times per month.
- 20 days of paid holiday per year and 6 days of sick leave per year, prorated.
- Weekends, Japanese national holidays, and other company-designated holidays off.
- Japanese social insurance, including health insurance, pension, workers’ compensation, unemployment insurance, and long-term care insurance.
- Housing allowance, retirement benefits, and rental car support.
- In-house software and language training programs.
Categories
About Woven by Toyota
Woven by Toyota exists to accelerate Toyota’s once-in-a-century transformation from a car company into a mobility company. Rooted in a legacy of reinvention, we carry forward a singular founding principle: to innovate for the benefit of others. Today, we are driving this transformation from two vantage points: redefining how vehicles are built, experienced, and connected to the world to realize a true software-defined vehicle, and reimagining how people, goods, information, and energy move and interact in society to empower individuals from all walks of life. Through these two lenses, we combine bold innovation with real-world execution, bridging the gap between present and future to help Toyota evolve its business and its role in society.