Member of Technical Staff - Training Platform
Prime Intellect4 months ago
Base Salary
$150k - $300k/yr
Responsibilities
- Design and operate Kubernetes-based training and inference orchestration across multi-cluster, multi-cloud GPU fleets.
- Build and maintain Helm charts and reproducible training stacks for trainers, inference servers, environment servers, and supporting services.
- Develop Python control-plane agents that monitor pods, report run state, and synchronize clusters with the platform.
- Implement scheduling and autoscaling for heterogeneous GPUs using KEDA, LeaderWorkerSet, taints and tolerations, and gang scheduling.
- Operate GitOps workflows, node-local model caches, checkpoint pipelines, shared storage, and observability systems for GPU clusters.
- Build developer-facing hosted-training features for job submission, live monitoring, logs, metrics, model and adapter management, and comparisons.
- Develop FastAPI backend services, REST APIs, real-time debugging tools, streaming logs, step-level metrics, and failure analysis workflows.
- Build product UI with Next.js, React, TypeScript, shadcn, Tailwind, tRPC, and TanStack Query.
- Interface with RL trainers, inference servers, and environment servers, and productize new model architectures, RL algorithms, and training modes.
Requirements
- Strong knowledge of modern AI stacks, open model families, LoRA, QLoRA, full fine-tuning, RLHF/RLAIF, and inference engines including vLLM, SGLang, and TensorRT-LLM.
- Understanding of GPU hardware tradeoffs involving H100, H200, B200, NVLink, interconnects, and memory hierarchy.
- Understanding of distributed training fundamentals including data, tensor, pipeline, and expert parallelism, NCCL, and multi-node scheduling.
- Strong Kubernetes operations experience with Helm, CRDs, operators, KEDA, gang scheduling, and GPU Operator, including production cluster debugging with kubectl.
- Cloud platform experience, preferably GCP, including GCS, GKE, Cloud Run, and Cloud Tasks.
- Experience with infrastructure automation using Helm, Terraform, and Ansible, plus GitOps workflows.
- Experience operating observability systems including Prometheus, Grafana, Loki, OpenTelemetry, and DCGM.
- Linux fundamentals including networking, namespaces, and performance tuning.
- Strong Python backend development with FastAPI, async programming, and SQLAlchemy, plus experience building Kubernetes control-plane agents.
- Modern frontend development with TypeScript, React, Next.js, Tailwind, and shadcn, sufficient to ship product surfaces end-to-end.
- Experience designing REST and tRPC APIs and building developer tools, dashboards, and live-monitoring UIs.
- Experience in platform or infrastructure development, ideally both, and interest in frontier AI infrastructure and open-source work.
Benefits
- Cash compensation of $150K–$300K with significant equity.
- Flexible work arrangement: remote or San Francisco office.
- Full visa sponsorship and relocation support.
- Professional development budget for courses and conferences.
- Regular team off-sites and conference attendance.
- Opportunity to contribute to frontier AI infrastructure, research, and open-source development.
Tech Stack
Categories
About Prime Intellect
Prime Intellect builds a full-stack platform for training frontier AI models, giving enterprises access to distributed compute and agentic training infrastructure to develop their own systems. The company operates as a platform and open research lab, with customers leveraging its infrastructure rather than building it in-house. Headquartered in San Francisco and privately held, it has raised a Series A round backed by investors including Founders Fund and NVIDIA.