AI Benchmark Engineer | Native Language Specialist - Czech - Remote
LILT (Production)6 months ago
Remote, CzechiaSenior
Responsibilities
- Design and engineer high-signal Terminal-Bench tasks for evaluating coding agents on multilingual software challenges.
- Create realistic Czech-language datasets, files, prompts, and task environments that preserve the target language.
- Develop reference implementations and reliable deterministic verifier scripts, using rubric-based judging only when necessary.
- Analyze execution logs and calibrate task difficulty from Easy to Very Hard across Haiku, Sonnet, and Opus model tiers.
- Participate in creation, human review, calibration review, audit, and automated quality-control processes.
- Evaluate multilingual text-processing behavior involving encoding, Unicode normalization, locale conventions, text I/O, toolchain interoperability, and safe string operations.
Requirements
- At least 5 years of industry experience in software engineering.
- A proven track record at leading technology companies and/or graduation from top-tier engineering universities.
- Native or near-native Czech fluency with deep knowledge of grammar, register, and phrasing, plus high English proficiency.
- Strong proficiency in Python, standard shell scripting, and data processing.
- Extensive experience with Terminal/CLI-based development workflows and familiarity with coding agents.
- Deep technical understanding of multilingual text-processing pitfalls, including encoding and decoding, Unicode normalization, locale-dependent conventions, non-Gregorian dates, and text I/O.
- For relevant languages, familiarity with bidirectional or RTL handling, font fallbacks, and rendering or typography in user interfaces or artifacts.
Benefits
- Remote independent-contractor work with flexible scheduling and no fixed hours.
- Competitive rates and prompt payments.
- Access to diverse AI and language-technology projects and a global expert community.
- Work availability fluctuates with project demand; hours are not guaranteed and the role is intended as supplemental income rather than a primary full-time job.
- Most accepted tasks require at least 2 hours per day or 10 hours per week.
- As a 1099 contractor, the worker receives no health insurance, paid time off, or retirement contributions and is responsible for personal tax obligations.
- Geographic restrictions may apply for regions subject to international embargoes or sanctions.