AI Benchmark Engineer | Native Language Specialist - Serbian - Remote

6 months ago

Remote, SerbiaSenior

Responsibilities

Design and engineer high-signal Terminal-Bench tasks for evaluating coding agents in multilingual software environments.
Create realistic datasets, files, prompts, and task assets that remain in Serbian and expose multilingual failure points.
Develop reference implementations and deterministic verifier scripts, using rubric-based judging only when necessary.
Run tasks against Haiku, Sonnet, and Opus model tiers, analyze execution logs, and calibrate difficulty from Easy to Very Hard.
Participate in creation, human review, calibration review, audit, and automated LLM-based quality-control processes.
Ensure benchmark tasks are fair, grammatically accurate, reliable, and technically valid.

At least 5 years of industry software engineering experience.
A proven track record at leading technology companies and/or graduation from top-tier engineering universities.
Native or near-native Serbian fluency with deep knowledge of grammar, register, and phrasing, plus high English proficiency.
Strong proficiency in Python, standard shell scripting, and data processing.
Extensive experience with Terminal/CLI-based development workflows and familiarity with coding agents.
Deep understanding of multilingual text-processing issues including encoding and decoding, Unicode normalization, locale conventions, text I/O, toolchain interoperability, and safe string operations.
For applicable languages, familiarity with bidirectional or RTL handling, font fallbacks, and rendering or typography in user interfaces or artifacts.

Remote, independent-contractor work with flexible scheduling and no fixed hours or check-ins.
Competitive rates and prompt payments.
Access to diverse AI and language-technology projects and a global expert community.
Work availability fluctuates with project demand; hours are not guaranteed and the role is not intended as a full-time primary-income position.
Most accepted tasks require at least 2 hours per day or 10 hours per week.
The role is a 1099 contractor position without health insurance, paid time off, retirement contributions, or employer-provided benefits.
Contractors are responsible for their own taxes, and geographic restrictions may apply in regions subject to international embargoes or sanctions.

501-1,000 employees