AI Benchmark Engineer | Native Language Specialist - Japanese - Remote
LILT (Production)6 months ago
Remote, JapanEntry Level
Responsibilities
- Design and build high-signal Terminal-Bench tasks that evaluate coding agents in multilingual software environments.
- Create realistic Japanese-language datasets, files, prompts, and task assets without relying on English translations.
- Develop reference implementations and reliable deterministic verifier scripts.
- Analyze execution logs and calibrate task difficulty from Easy to Very Hard across Haiku, Sonnet, and Opus model tiers.
- Participate in task creation, human review, calibration review, audit, and automated LLM-based quality checks.
- Ensure grammatical accuracy, fairness, benchmark integrity, and robust handling of encoding, Unicode, locale, and text-processing edge cases.
Requirements
- At least 1 year of experience in software engineering or prompt engineering.
- Proven track record at leading technology companies and/or graduation from top-tier engineering universities.
- Native or near-native Japanese fluency with deep knowledge of Japanese grammar, register, and phrasing, plus high English proficiency.
- Strong proficiency in Python, standard shell scripting, and data processing.
- Extensive Terminal/CLI-based development experience and familiarity with coding agents.
- Deep technical understanding of encoding and decoding, Unicode normalization, locale-dependent conventions, text I/O, toolchain interoperability, and safe string operations.
- For applicable languages, knowledge of bidirectional or RTL handling, font fallbacks, and rendering or typography in user interfaces or artifacts.
Benefits
- Remote, flexible independent-contractor work with no fixed hours or micromanagement.
- Competitive rates and prompt payments.
- Access to diverse AI and language-technology projects and a global expert community.
- 1099 contractor arrangement with no health insurance, paid time off, retirement contributions, or guaranteed hours.
- Most accepted tasks require at least 2 hours per day or 10 hours per week, and work availability fluctuates with project demand.
- Contractors are responsible for their own tax obligations, and geographic restrictions may apply in embargoed or sanctioned regions.