Senior Machine Learning Engineer
Ambience Healthcare2 months ago
Base Salary
$225k - $300k/yr
Responsibilities
- Design and own evaluation pipelines for LLM and agentic systems using automated graders, regression testing, production feedback, and human evaluation.
- Diagnose high-impact production failure modes and test improvements involving prompting, retrieval, context, routing, data, fine-tuning, and model or system interventions.
- Develop production agentic AI systems with tool use, retrieval, context and state management, routing, orchestration, tracing, and failure recovery.
- Turn production failures and user feedback into improved datasets, evaluations, and model behavior through active learning and systematic iteration.
- Translate recent research in LLMs, agents, NLP, speech, and multimodal AI into practical experiments.
- Own AI systems end-to-end across models, data, evaluation, orchestration, serving, and observability.
- Write code, inspect traces, analyze failures, and debug production systems while working with clinicians, product managers, and engineers.
Requirements
- At least 5 years of experience in production machine learning, research engineering, or applied AI.
- Experience building a consequential production AI system or materially improving model behavior in production.
- Strong understanding of modern LLMs, transformers, and production AI systems.
- Experience designing evaluations for LLMs, agents, or other complex AI systems and translating quality problems into measurable datasets and experiments.
- Experience building production systems involving multiple models, tools, retrieval, context, state, routing, or orchestration.
- Understanding of reliability and failure modes in complex AI workflows.
- Proficiency in Python and modern machine-learning frameworks; PyTorch is preferred.
- Comfort with deployment, observability, and containerized systems, and continued hands-on production coding and debugging.
- Experience building high-quality datasets and feedback loops, including using production failures, user feedback, and active learning.
- Ability to collaborate with clinicians, product managers, and engineers and communicate complex AI concepts to diverse audiences.
- Experience with realtime voice, conversational AI, or multimodal systems is a plus.
- Experience with fine-tuning, post-training, or model adaptation is a plus.
- Prior experience in healthcare, clinical AI, or other regulated, high-stakes industries is a plus.
- Experience interviewing or mentoring ML engineers is a plus.
- Open-source contributions to ML, agent, or evaluation tooling are a plus.
Benefits
- Hybrid engineering role with onsite work at the San Francisco office three days per week; the company also describes a remote-friendly culture.
- Comprehensive medical, dental, and vision coverage for employees and dependents.
- 401(k) with company match of up to 3% of base salary.
- Full equipment provisioning.
- Parental leave.
- Annual company-wide and team off-sites, team lunches, and all-hands gatherings with travel, lodging, and meals covered.
- Flexible time off with no annual cap, company-wide holidays, and an annual holiday shutdown from December 24 through January 1.
Categories
About Ambience Healthcare
Ambience Healthcare builds an AI platform for clinical documentation, medical scribing, and coding that reduces administrative work for clinicians in outpatient, emergency, and inpatient settings. It sells enterprise software to health systems and clinics, with real-time, coding‑aware notes and patient summaries that integrate with Epic, Oracle Cerner, athenahealth, and other EHRs. Founded in 2020 and headquartered in San Francisco, it supports 100+ specialties across North America.