3 hours ago
Base Salary
$159k - $219k/yr
Responsibilities
- Work with client infrastructure teams on GKE training platforms, GPU node pools, job queueing, scheduling, quota, and capacity management.
- Diagnose and fix training platform failures at their root cause.
- Run, scale, and debug multi-node distributed training with PyTorch, FSDP, DeepSpeed, Megatron or NeMo-style approaches, and Ray Train.
- Tune multi-node GPU networking using RDMA, GPUDirect, and NCCL, including GB200 or H200 topologies.
- Profile jobs and remove bottlenecks across compute, memory, input pipelines, and interconnects.
- Design and run cluster and training-stack performance benchmarks measuring throughput, scaling efficiency, and cost per run.
- Build model evaluation harnesses and maintain telemetry, monitoring, and end-to-end pipeline tests.
- Own platform issues proactively, communicate status and risks, and explain technical decisions to client engineers and scientists.
Requirements
- 5+ years of engineering experience with recent hands-on work on GPU training infrastructure and large-scale training.
- Strong Python software engineering skills, including production-quality code, testing, and code review.
- Experience with Kubernetes on Google Cloud, GKE GPU node pools, job queueing such as Kueue, and capacity and quota management.
- Hands-on experience running and debugging multi-node distributed training jobs with PyTorch and Ray Train.
- Experience with RDMA, GPUDirect, NCCL tuning, and multi-node GB200 or H200 topology.
- Experience designing performance benchmarks and model evaluations and using their results to guide decisions.
- Mastery of shell scripting and familiarity with Google Gemini, GPT-4, or LLaMA, prompt engineering, and fine-tuning.
- Experience with vector databases, semantic search, Google Cloud Vertex AI, scalable software architectures, and orchestration frameworks such as LangChain or LlamaIndex.
- Strong software engineering fundamentals, including clean maintainable code and understanding of the full software development lifecycle.
- Visible ownership, communication, accountability, and ability to explain system behavior and design decisions in depth.
- Healthcare or life-sciences data experience, HIPAA, and observability and telemetry for training platforms are nice to have.
Benefits
- Comprehensive health insurance.
- Paid vacation/PTO, holidays, and sick leave.
- Parental leave and bereavement leave.
- 401(k) employer match.
- Employee referral bonuses.
- Standard hiring verification practices may include background and employment checks.
Tech Stack
Categories
About Egen
Egen is a privately held IT services firm that designs and builds cloud, data, analytics, AI, and platform solutions for enterprises and public-sector clients. It delivers consulting, engineering, and managed services across major ecosystems, including Google Cloud and Salesforce. Founded in 2000 and headquartered in Naperville, Illinois, Egen operates across North America, Europe, and India.