(Staff / Senior Staff) AI Infra Engineer, Inference Infrastructure
Together AI4 hours ago
Base Salary
$250k - $320k/yr
Responsibilities
- Operate and improve the production provisioning state machine covering host lifecycle, discovery, inference bring-up, GPU driver and CUDA validation, health checks, decommissioning, and RMA.
- Diagnose fleet-scale state-transition failures and extend automated self-healing with new detection and remediation plugins.
- Own provisioning and fleet-health SLOs and error budgets, lead incident response, and implement systemic fixes for idempotency, retry, rollback, and drift failures.
- Partner with inference and ML platform teams to translate cluster topology, interconnect, scheduling, and serving failure modes into platform requirements.
- Develop techniques, tools, and architecture that improve inference performance, latency, throughput, and efficiency.
- Build observability and troubleshooting tools and implement solutions for bottlenecks and instability.
- Lead cross-functional reliability planning across infrastructure, ML platform, and product teams.
Requirements
- Production experience operating provisioning or host-lifecycle pipelines and troubleshooting failed state transitions at fleet scale.
- Deep production-reliability experience with large-scale distributed systems; GPU or ML-serving infrastructure experience is strongly preferred.
- Hands-on experience troubleshooting InfiniBand and RDMA networking, including IB port counters and subnet manager logs.
- Experience with multi-tenant model serving, LoRA or adapter-based serving, and differentiated SLOs across model pools.
- A demonstrated record of eliminating operational toil through automation and self-built tooling.
- Hands-on experience with Kubernetes controllers, CRD patterns, reconciliation loops, and node-level health or remediation systems.
- Comfort with event-driven architectures, backpressure, circuit breaking, and integrated metrics, logs, and traces.
- Strong software-engineering skills with experience writing and shipping production code.
Benefits
- Full-time position with competitive compensation, startup equity, health insurance, and other benefits.
- US base salary range of $250,000-$320,000 plus equity and benefits.
Tech Stack
Categories
DevOpsSite Reliability
About Together AI
Together AI builds an AI-native cloud platform for developers, offering high-performance inference, fine-tuning/model shaping, and large-scale pre-training on on-demand GPU clusters with APIs and managed services. It emphasizes open-source models that teams can run and adapt, and also provides infrastructure for decentralized and scalable workloads. Founded in 2022 and headquartered in San Francisco, it is privately held and reports notable customers including Cursor, ElevenLabs, Salesforce, and Zoom.