about 6 hours ago
New York, NY, USAMid Level / Senior
Base Salary
$180k - $250k/yr
Responsibilities
- Design and implement internal tools for prompt management, evaluation, experimentation, and model iteration.
- Create systems that enable rapid testing, debugging, and comparison of model outputs.
- Develop frameworks for benchmarking, regression testing, and monitoring AI system performance.
- Build pipelines and tooling to support data collection, labeling, and feedback loops.
- Partner with engineering teams to ensure AI features are reliable and maintainable in production.
- Identify bottlenecks in the development process and build systems to increase developer velocity.
Requirements
- 3–7+ years of software engineering experience, ideally in developer tools or infrastructure.
- Strong proficiency in Python and experience building backend systems.
- Hands-on experience with LLMs, APIs, and modern AI workflows.
- Experience in building systems for experimentation, evaluation, or data pipelines.
- Strong systems thinking with the ability to design scalable tools.
- Excellent communication skills to collaborate with engineers and researchers.
- Proactive and resourceful, with a focus on building foundational systems.