Machine Learning Engineer - ML Training Platform

5 months ago

Responsibilities

Design multi-cloud resource management systems across AWS, GCP, and Azure using Pulumi or Terraform.
Build dynamic scaling, state synchronization, and concurrent operations for hundreds of heterogeneous nodes.
Architect fault-tolerant infrastructure for distributed machine-learning training, including GPU clusters, NVIDIA runtime support, S3 checkpointing, dataset streaming, health monitoring, and retry strategies.
Manage large datasets, data flow, model-training jobs, and resilient distributed execution across workers.
Build systems for bandwidth shaping, latency injection, packet loss, dynamic node churn, and heterogeneous network connectivity.
Develop infrastructure orchestration and service integration for continuous experimentation and large-scale model training.
Implement monitoring, observability, performance profiling, incident response, and reliability practices.

Ideally 5+ years of work experience in infrastructure, platform engineering, distributed systems, or related areas.
Production experience with infrastructure-as-code and multi-cloud deployments using Pulumi, Terraform, or CloudFormation.
Experience with lifecycle orchestration, self-healing systems, Docker, Kubernetes, EKS, GPU workloads, and heterogeneous clusters at scale.
Deep understanding of distributed training workflows, checkpointing, data sharding, model versioning, long-running job orchestration, decentralized networking, NAT traversal, traffic shaping, and bandwidth constraints.
Strong Python engineering skills, including asyncio, concurrency, retry logic, cloud SDKs, and CLI tooling.
Hands-on experience with observability, SRE practices, Prometheus, Grafana, performance profiling, monitoring, and incident response.
Experience in a startup environment focused on microservices orchestration or a big-tech background.
Strong understanding of multi-cloud infrastructure and distributed training systems.
High attention to detail, teamwork, and strong motivation to join the company.

1-10 employees

Pluralis is developing a protocol that facilitates collaborative training and ownership of foundation models.