(Senior/Junior AI Inference Reliability Engineer, Compute Infra
Together AI2 days ago
Base Salary
$160k - $280k/yr
Responsibilities
- Operate and improve the production provisioning state machine across host discovery, inference bring-up, GPU driver/CUDA validation, health checks, and decommissioning.
- Diagnose production failures involving InfiniBand/RDMA, GPU Xid variants, model-serving faults, Kubernetes controllers, and fleet-scale state transitions.
- Extend automated self-healing through new detection and remediation plugins.
- Own provisioning and fleet-health SLOs and error budgets, lead incident response, and drive systemic fixes for idempotency, retries, rollbacks, and drift.
- Partner with inference and ML platform teams to incorporate real-world cluster, interconnect, and scheduling failure modes into platform requirements.
- Build tools and architectural improvements for inference performance, latency, throughput, capacity efficiency, observability, and stability.
- Lead cross-functional reliability priorities across infrastructure, ML platform, and product teams.
Requirements
- Production experience operating provisioning or host-lifecycle pipelines at fleet scale.
- Deep production-reliability experience with large-scale distributed systems; GPU or ML-serving infrastructure experience is preferred.
- Hands-on experience troubleshooting InfiniBand/RDMA issues, including IB port counters and subnet manager logs.
- Familiarity with multi-tenant model serving, LoRA or adapter-based serving, and differentiated SLOs across model pools.
- A track record of eliminating operational toil through automation and self-built tooling.
- Experience with Kubernetes controller and CRD patterns, reconciliation loops, and node-level health or remediation systems.
- Comfort with event-driven architectures, backpressure, circuit breaking, and built-in metrics, logs, and traces.
- Strong software-engineering skills and experience shipping production code.
Benefits
- Competitive compensation, startup equity, health insurance, and other competitive benefits.
- Full-time position with a US base salary range of $160,000-$280,000 plus equity and benefits.
Tech Stack
Categories
DevOpsSite Reliability
About Together AI
Together AI builds an AI-native cloud platform for developers, offering high-performance inference, fine-tuning/model shaping, and large-scale pre-training on on-demand GPU clusters with APIs and managed services. It emphasizes open-source models that teams can run and adapt, and also provides infrastructure for decentralized and scalable workloads. Founded in 2022 and headquartered in San Francisco, it is privately held and reports notable customers including Cursor, ElevenLabs, Salesforce, and Zoom.