AI Quality & Evaluation Engineer, AI product
Woven by Toyota5 months ago
Tokyo, JapanMid Level
Responsibilities
- Define quality dimensions for AI products, including accuracy, consistency, safety, fairness, and UX.
- Conduct scenario-based testing, exploratory testing, and red teaming.
- Analyze LLM outputs to identify behavioral trends and failure patterns.
- Design evaluation processes using user feedback, logs, ratings, and inquiries.
- Standardize and automate reusable, scalable quality evaluation workflows.
- Collect and aggregate evaluation data and establish recurring quality checks.
- Build and operate quality feedback loops with development, MLOps, and data engineering teams.
- Document quality issues and risks and communicate them to relevant teams.
Requirements
- At least 3 years of QA or testing experience with products that use LLMs.
- Experience designing tests, including defining test perspectives and creating test cases.
- Ability to establish clear judgment criteria for systems with ambiguous specifications or non-unique correct answers.
- Experience automating test or evaluation processes.
- Experience collaborating with development teams to drive quality improvements.
- Business-level proficiency in both English and Japanese.
- Experience with scenario testing, red teaming, user-feedback and log analysis, and Japanese-language natural language processing is preferred.
Benefits
- Hybrid work arrangement with flexible working hours.
- Business trips to Woven City several times per month.
- 20 days of paid holiday per year and 6 days of sick leave per year, prorated.
- Weekends, Japanese national holidays, and other company-designated holidays off.
- Japanese social insurance, including health insurance, pension, workers’ compensation, unemployment insurance, and long-term care insurance.
- Housing allowance, retirement benefits, and rental car support.
- In-house software and language training programs.
Categories
About Woven by Toyota
Woven by Toyota develops software for automated driving, advanced driver assistance, and software-defined vehicles, primarily for Toyota and its partners. Its products include Arene (a vehicle software platform), AD/ADAS technologies, and the Woven City test course, supported by cloud and AI infrastructure. Founded in 2023 and headquartered in Chuo-ku, Tokyo, it operates as a Toyota Motor Corporation subsidiary focused on vehicle software and mobility systems.