about 1 month ago
New York, NY, USAMid Level / Senior
Base Salary
$180k - $250k/yr
Responsibilities
- Design and implement internal tools for prompt management, evaluation, experimentation, and model iteration.
- Create systems that enable rapid testing, debugging, and comparison of model outputs across different approaches.
- Develop frameworks for benchmarking, regression testing, and monitoring AI system performance.
- Build pipelines and tooling to support data collection, labeling, and feedback loops for model improvement.
- Partner with engineering teams to ensure AI features are reliable, observable, and easy to maintain in production.
- Identify bottlenecks in the development process and build systems that increase developer velocity and confidence.
Requirements
- 3–7+ years of software engineering experience, ideally working on developer tools, infrastructure, or platform teams.
- Strong proficiency in Python and experience building backend systems and internal tooling.
- Hands-on experience working with LLMs, APIs (e.g., OpenAI, Anthropic), and modern AI workflows.
- Experience building systems for experimentation, evaluation, or data pipelines.
- Strong systems thinking with the ability to design tools that scale across teams.
- Highly pragmatic with a focus on improving developer velocity and real-world outcomes.
- Strong communicator who can work closely with engineers, researchers, and product teams.
- Proactive, resourceful, and excited about building foundational systems in a fast-moving environment.