Senior or Staff Software Engineer, Inference / Compute Infrastructure Engineering
Together AI19 days ago
Remote, IndiaSenior / Staff+
Responsibilities
- Design and implement a versioned provisioning state machine covering physical host discovery, inference bring-up, GPU driver and CUDA setup, health validation, decommissioning, and RMA.
- Build declarative self-service APIs and a Kubernetes-native control plane for requesting, scaling, and tearing down inference clusters.
- Automate detection of degraded or failed nodes, safe draining, repair or replacement, and reintroduction of healthy capacity.
- Ensure provisioning reliability through idempotency, retries, rollback, drift detection, and durable workflow execution.
- Partner with inference and ML platform teams to encode topology, interconnect, and scheduling constraints as platform abstractions.
- Improve GPU utilization through defragmentation, rebalancing, scheduling, bin-packing, and right-sizing logic.
- Operate and support the software in production using strong typing, automated tests, code review, versioning, and CI/CD for infrastructure code.
Requirements
- Strong software engineering experience with Go, Python, Rust, or a similar language, including writing and testing production software.
- Experience with durable workflow orchestration tools such as Temporal, Cadence, or equivalent.
- Experience building control planes or orchestration systems with state modeling and reconciliation, such as Kubernetes controllers, operators, custom reconciliation loops, or workflow engines.
- Experience designing event-driven systems using message queues, event streams, or pub/sub rather than polling or cron-driven scripts.
- Experience building internal platforms or APIs consumed by other engineering teams, with attention to developer experience.
- Preferred experience with bare-metal provisioning technologies such as PXE, iPXE, Redfish, IPMI, or BMC.
- Preferred networking experience with VLANs, BGP, or fabric design.
- Preferred experience with GPU or accelerator infrastructure and GPU cluster software such as NCCL, CUDA, InfiniBand, or RoCE.
- Experience at a hyperscaler, GPU cloud, or datacenter-scale infrastructure organization is preferred.
- Systems programming experience in Rust or Go is preferred.
Tech Stack
About Together AI
Together AI builds an AI-native cloud platform for developers, offering high-performance inference, fine-tuning/model shaping, and large-scale pre-training on on-demand GPU clusters with APIs and managed services. It emphasizes open-source models that teams can run and adapt, and also provides infrastructure for decentralized and scalable workloads. Founded in 2022 and headquartered in San Francisco, it is privately held and reports notable customers including Cursor, ElevenLabs, Salesforce, and Zoom.