Applied AI Engineer
OpenLoop Health3 hours ago
Remote, United StatesSenior / Staff+
Responsibilities
- Build and operate production LLM-based agents and assistants used by real users.
- Develop agent runtime and orchestration capabilities including tool calling, context, handoffs, retries, timeouts, failure handling, and sandboxing.
- Create evaluation datasets, test harnesses, regression tests, and AI-graded evaluations.
- Manage model providers, version upgrades, and provider fallback strategies.
- Build production tracing, behavior monitoring, and AI cost attribution.
- Implement retrieval, context design, and prompt engineering to ground agent responses in company data.
- Collaborate with operations teams to determine which workflow steps agents can perform and when to escalate to people.
- Protect PHI, control agent access, and defend against prompt injection and other misuse.
- Build on GCP and partner with Data Platform teams on governed data foundations.
- Contribute to code and design reviews, mentor engineers newer to AI, and explain AI trade-offs to technical and non-technical stakeholders.
Requirements
- Bachelor's degree in Computer Science, Engineering, or a related technical field.
- 5 to 15+ years of software engineering experience.
- Hands-on experience building and running production LLM systems relied on by real users.
- Deep experience in at least one of agent runtime, evaluation, retrieval, or observability and cost.
- Experience measuring AI system performance with evaluation data.
- Strong production engineering skills across testing, debugging, services, APIs, deployment, and on-call.
- Experience handling sensitive data with a security-first mindset.
- Clear communication with technical and non-technical partners and comfort working in greenfield environments.
- Preferred qualifications include healthcare, finance, or insurance experience; GCP production experience; model migration experience; classic machine learning experience; internal platform development; and technical team leadership for Senior Staff candidates.
Benefits
- Medical, dental, and vision plans.
- Flexible Spending Accounts and Health Savings Accounts.
- Flexible PTO.
- 401(k) with company match.
- Life insurance, pet insurance, and other benefits.
- Salaried positions are eligible for the listed benefits.
Tech Stack
Categories
About OpenLoop Health
OpenLoop Health provides white-label telehealth infrastructure and clinical operations for digital health companies, health plans, and health systems. Its end-to-end services span provider staffing and licensing, payer coverage, a virtual care platform with AI, pharmacy and diagnostics coordination, RPM fulfillment, and revenue cycle management, enabling nationwide virtual care delivery. Founded in 2020 and headquartered in Des Moines, Iowa, the company is privately held and supports programs operating across all 50 U.S. states.