AI Benchmark Engineer | Native Language Specialist - Chinese (Taiwan) - Remote
LILT (Production)10 hours ago
Remote, TaiwanSenior
Responsibilities
- Design and engineer high-signal Terminal-Bench tasks for evaluating coding agents on multilingual software challenges.
- Build realistic task environments, datasets, and files that remain in the target language.
- Create native-language prompts and identify failure points in AI systems.
- Develop reference implementations and reliable deterministic verifier scripts.
- Analyze execution logs and calibrate task difficulty from Easy to Very Hard using standard Terminal-Bench configurations across Haiku, Sonnet, and Opus.
- Participate in creation, human review, calibration review, audit, and automated LLM-based quality checks.
- Ensure benchmark fairness, grammatical accuracy, reproducibility, and integrity.
Requirements
- 5+ years of industry experience in software engineering.
- Proven track record at leading technology companies and/or graduation from top-tier engineering universities.
- Native or near-native fluency in Chinese (Taiwan), with deep understanding of grammar, register, and phrasing, plus high English proficiency.
- Strong proficiency in Python, standard shell scripting, and data processing.
- Extensive experience with Terminal/CLI-based development workflows and familiarity with coding agents.
- Deep understanding of multilingual text-processing issues, including encoding and decoding, Unicode normalization, locale-dependent conventions, text I/O, toolchain interoperability, and safe string operations.
- For applicable languages, familiarity with bidirectional or RTL handling, font fallbacks, and rendering or typography in user interfaces or artifacts.
- Ability to complete a GenAI assessment and provide an updated CV in English.
Benefits
- Remote, flexible independent-contractor work with control over schedule and workload.
- Competitive rates and prompt payments.
- Access to diverse AI and language-technology projects and a global expert community.
- Portfolio-building opportunities across innovative projects.
- Work availability fluctuates with project demand; hours are not guaranteed and the role is not intended as a full-time or primary-income position.
- Most accepted tasks require at least 2 hours per day or 15–20 hours per week.
- The role is a 1099 contractor position without health insurance, paid time off, retirement contributions, or other employee benefits.
- Contractors are responsible for their own tax obligations, and geographic restrictions may apply in regions subject to international embargoes or sanctions.
About LILT (Production)
LILT builds an enterprise translation and localization platform that combines AI models with human linguists to produce multilingual content for businesses and governments. The company also provides data labeling and evaluation services for AI developers, plus managed translation services integrated with common content and support tools. Founded in 2015 and headquartered in San Francisco, it is privately held and works with global organizations that need at-scale, domain-specific localization.