about 4 hours ago
Remote, United States
Senior
H1B Sponsor
Base Salary
$136k - $170k/yr
Responsibilities
- Design and ship a robust, end-to-end AI evaluation framework.
- Define and instrument key performance metrics for AI agents.
- Build and maintain evaluation datasets and automated scoring pipelines.
- Architect and implement reusable agent infrastructure.
- Build and scale retrieval infrastructure and RAG pipelines.
- Make informed build vs. buy decisions for AI tools and frameworks.
- Own projects end-to-end and drive them to completion.
- Collaborate with engineering leads to inform technical direction.
Requirements
- 5+ years of professional software engineering experience in production AI/ML systems.
- Deep hands-on experience with LLM-based systems and evaluation metrics.
- Proven ability to work with data and understand statistics.
- Experience building and operating agentic AI systems in production.
- Strong command of AI evaluation frameworks.
- Production-grade Python engineering skills.
Benefits
- Medical, dental, and vision insurance.
- Life, AD&D, and disability insurance.
- Paid parental leave and paid time off.
- Commuter and parking accounts.
- Internet and phone stipend.
- 401(k) retirement plan and financial planning support.
- Learning and development budget.
Tech Stack
AWSDatadogGoogle Cloud PlatformMLflowPythonTypeScript
Categories
AI & MLData Science