AI Benchmark Engineer | Native Language Specialist - Arabic (Saudi Arabia) - Remote
LILT (Production)10 hours ago
Remote, Saudi ArabiaSenior
Responsibilities
- Design and engineer Terminal-Bench tasks that evaluate coding agents on multilingual software challenges.
- Build realistic task environments, datasets, and files in Arabic without relying on English translations.
- Identify AI failure points through native-language prompting and translation work.
- Develop reference implementations and highly reliable deterministic verifier scripts.
- Analyze execution logs and calibrate task difficulty from Easy to Very Hard across Haiku, Sonnet, and Opus model tiers.
- Participate in creation, human review, calibration review, audit, and automated quality-control processes.
- Ensure benchmark tasks are fair, grammatically accurate, reproducible, and technically robust.
Requirements
- 5+ years of industry experience in software engineering.
- Proven experience at leading technology companies and/or graduation from top-tier engineering universities.
- Native or near-native Arabic fluency with deep knowledge of grammar, register, and phrasing, plus high English proficiency.
- Strong proficiency in Python, standard shell scripting, and data processing.
- Extensive experience with Terminal/CLI-based development workflows and familiarity with coding agents.
- Deep understanding of encoding and decoding, Unicode normalization, locale-dependent conventions, text I/O, toolchain interoperability, safe string operations, and Arabic bidirectional/RTL handling, font fallbacks, and rendering or typography issues.
- Reliable availability and commitment to on-time delivery after accepting tasks.
Benefits
- Independent contractor flexibility to choose when and how much to work.
- Remote freelance work with no fixed hours, check-ins, or micromanagement.
- Competitive rates and prompt payments.
- Access to diverse AI and language-technology projects and a global expert community.
- Work availability fluctuates with project demand; this is intended as supplemental income rather than a full-time primary role.
- Most tasks require at least 2 hours per day or 15–20 hours per week, though hours are not guaranteed.
- This is a 1099 contractor role without health insurance, paid time off, or retirement contributions; contractors are responsible for their own taxes.
- Geographic restrictions may apply in regions subject to international embargoes or sanctions.
About LILT (Production)
LILT builds an enterprise translation and localization platform that combines AI models with human linguists to produce multilingual content for businesses and governments. The company also provides data labeling and evaluation services for AI developers, plus managed translation services integrated with common content and support tools. Founded in 2015 and headquartered in San Francisco, it is privately held and works with global organizations that need at-scale, domain-specific localization.