AI Benchmark Engineer | Native Language Specialist - German - Remote
LILT (Production)6 months ago
Remote, GermanyEntry Level
Responsibilities
- Design and engineer high-signal Terminal-Bench tasks for evaluating coding agents in multilingual software scenarios.
- Build realistic task environments using datasets and files in the target language.
- Create native-language prompts and identify failure points in AI systems.
- Develop reference implementations and deterministic verifier scripts.
- Run benchmark configurations across Haiku, Sonnet, and Opus model tiers and calibrate task difficulty from Easy to Very Hard.
- Analyze execution logs and participate in creation, review, calibration, and audit quality-control processes.
- Evaluate multilingual handling of encoding, Unicode, locale conventions, text I/O, toolchain interoperability, and safe string operations.
Requirements
- At least 1 year of industry experience in software engineering or prompt engineering.
- A proven track record at leading technology companies and/or graduation from top-tier engineering universities.
- Native or near-native fluency in German with deep knowledge of grammar, register, and phrasing, plus high English proficiency.
- Strong proficiency in Python, standard shell scripting, and data processing.
- Extensive experience with Terminal/CLI-based development workflows and familiarity with coding agents.
- Deep technical understanding of multilingual text-processing issues, including encoding and decoding, Unicode normalization, locale-dependent conventions, text I/O, and interoperability.
- For applicable languages, familiarity with bidirectional or RTL handling, font fallbacks, and rendering or typography in interfaces or artifacts.
Benefits
- Remote, independent-contractor work with flexible scheduling and no fixed hours.
- Competitive rates and prompt payments.
- Access to diverse AI and language-technology projects and a global expert community.
- Project availability may fluctuate, and hours are not guaranteed; most accepted tasks require at least 2 hours per day or 10 hours per week.
- This is a 1099 contractor role without health insurance, paid time off, or retirement contributions, and contractors are responsible for their own taxes.
- Geographic restrictions may apply in regions subject to international embargoes or sanctions.