4 days ago
Remote, Spain +8 moreSenior
Responsibilities
- Review coding tasks derived from real GitHub issues and pull requests.
- Assess whether problem statements and success criteria are clear, complete, and technically solvable.
- Evaluate unit tests for correctness, coverage, robustness, and appropriate implementation flexibility.
- Identify flaky tests, missing dependencies, version conflicts, and environment issues.
- Determine whether tasks can be reliably reproduced across environments.
- Assess the real-world difficulty and complexity of each task.
- Provide clear recommendations on whether tasks should be accepted, improved, or excluded.
Requirements
- At least 3 years of professional software engineering experience.
- Strong experience working with large, multi-file codebases.
- Experience reviewing pull requests, debugging issues, and maintaining production code.
- Strong understanding of unit testing and test coverage.
- Ability to evaluate whether tests correctly validate solutions without unnecessarily restricting implementation approaches.
- Experience with dependency management, environment setup, and reproducibility.
- Strong understanding of Git and GitHub-based development workflows.
- Ability to analyze complex technical problems and provide clear written feedback.
- Preferred experience includes open-source contributions or maintenance, SWE-Bench or similar benchmarks, major Python open-source projects, Docker, pip, conda, dependency pinning, test fixtures, test isolation, property-based testing, technical assessment design, and AI/ML evaluation or benchmark development.
Benefits
- Remote work arrangement.
- Part-time, project-based consulting engagement.