Staff Site Reliability Engineer
Hippocratic AI2 months ago
Responsibilities
- Design and build the GPU management and scheduling platform for approximately 30 models running across heterogeneous hardware.
- Build metrics pipelines and decision logic for GPU utilization, admission control, load shedding, and capacity management.
- Implement autoscaling that adjusts model replicas based on real-time demand and utilization.
- Develop cloud orchestration systems and operators in Python and Go.
- Architect and operate scalable, fault-tolerant, and secure production systems on AWS, GCP, or Azure.
- Build infrastructure automation and deployment pipelines using Terraform and GitLab CI/CD.
- Maintain monitoring, logging, and alerting systems for platform reliability and performance.
- Develop and enforce security and compliance policies for a healthcare AI platform.
- Partner with engineers and research scientists to diagnose and resolve infrastructure, deployment, and operational issues.
- Mentor engineers and raise the team's technical standards.
Requirements
- 10+ years of professional experience across site reliability or DevOps engineering and software engineering.
- A Computer Science degree is required.
- Strong software engineering fundamentals, including building orchestration and scheduling systems in Python and/or Go.
- Experience designing systems that interpret operational metrics and drive control loops such as autoscaling, load shedding, or admission control.
- Deep experience with infrastructure automation and GitLab CI/CD, Terraform, or similar tools.
- Hands-on production experience with AWS, GCP, or Azure.
- Strong knowledge of Docker and Kubernetes.
- Experience with monitoring and logging stacks such as ELK, Grafana, or Datadog.
- Familiarity with secrets management and security tooling such as HashiCorp Vault, AWS KMS, or Azure Key Vault.
- Strong problem-solving, communication, interpersonal, independent-working, and collaborative skills.
- Experience managing GPU fleets or scheduling workloads across heterogeneous accelerators is preferred.
- Familiarity with ML inference serving and model deployment tools such as Triton, KServe, or Ray Serve is preferred.
- Experience with Kubernetes autoscaling internals, including HPA, VPA, custom metrics, and custom controllers, is preferred.
- Experience implementing HIPAA and SOC 2 compliance is preferred.
- Experience operating in an HPC environment is preferred.
- A bachelor's or master's degree in Computer Science, Computer Engineering, or a related field is listed as a nice-to-have qualification.
Tech Stack
Categories
Site Reliability
About Hippocratic AI
Hippocratic AI builds a safety-focused large language model and AI agents for healthcare workflows, used by health systems for patient outreach, post-discharge follow-up, and chronic-care management. It licenses its platform and tools to providers to automate and scale clinical support tasks while meeting health-system requirements. Founded in 2023 and headquartered in Palo Alto, the company is privately held.