18 hours ago
Remote, Czechia +4 moreSenior
Responsibilities
- Build realistic simulated developer environments with codebases, infrastructure, tickets, documentation, conversations, and development history.
- Design challenging tasks from intermediate environment states and define clear, solvable success criteria.
- Write functional and integration tests that accept valid agent solutions and reject incorrect ones.
- Review agent solutions, analyze failures, and refine tasks and tests based on QA feedback.
- Create evaluations that reveal meaningful differences between strong and weak AI coding agents.
Requirements
- At least 5 years of software development experience.
- Experience writing functional and integration tests.
- Proficiency with Python and FastAPI.
- Proficiency with JavaScript or TypeScript and React.
- Experience with Docker, Postgres, Kafka, and Redis.
- English proficiency at B2 level or higher.
- A master’s degree in computer science, software engineering, data science/data analytics, artificial intelligence/machine learning, computational linguistics/natural language processing, information systems, or a related field.
- A bachelor’s degree is accepted with 5 years of experience in the field.
- At least 3 years of professional experience in related QA automation/testing or cybersecurity roles is stated for those roles.
Benefits
- Remote, part-time freelance project that can fit around other professional or academic commitments.
- Project-based participation with payment per accepted task, up to the equivalent of $50/hr depending on qualification tier and efficiency.
- Opportunity to work on advanced AI projects, build portfolio experience, and influence how future AI models understand and communicate in the field.
Tech Stack
Categories
About Mindrift
Mindrift builds an expert-sourcing platform that connects domain specialists to project-based work training and evaluating generative AI models, including supervised fine-tuning, RLHF, evaluation, and red-teaming. It is built and operated by Toloka, part of Nebius Group, and run from Amsterdam, Netherlands. Work is fully remote and freelance, serving global technology companies developing and improving large AI systems.