Principal ML Engineer
Electronic Arts3 days ago
Base Salary
$227k - $345k/yr
Responsibilities
- Architect and build shared multimodal training input pipelines from certified game capture through storage, decoding, sharding, sampling, packing, batching, caching, prefetching, and host-to-device delivery.
- Define schemas, data contracts, encodings, compression, dataset versioning, lineage, quality standards, coverage, acceptance criteria, and reproducible splits.
- Build deterministic loaders and checkpoint-safe iteration that recover from worker failures, hardware failures, corrupt data, elastic restarts, and cluster or parallelism changes.
- Design reproducible dataset mixtures and balanced batches across modalities, resolutions, sequence lengths, game titles, and capture conditions.
- Benchmark and optimize throughput, data-wait time, step time, accelerator utilization, MFU, scaling efficiency, restart time, reliability, and cost across thousands of accelerators.
- Diagnose bottlenecks and intermittent failures across storage, network I/O, CPU, memory, decoding, preprocessing, accelerators, distributed workers, and data.
- Integrate compilers, runtimes, optimized kernels, low-precision training, and hardware-accelerated media processing across distributed parallelism strategies.
- Lead training-pipeline architecture reviews, mentor engineers, and translate research and data-access requirements into scalable systems.
Requirements
- 20+ years of development experience, including ownership of production systems for distributed-training input and measurable improvements in performance, reliability, recovery, or cost.
- Expertise building custom PyTorch training and ingestion systems or adapting TorchTitan, JAX/XLA with MaxText, Megatron Core/NeMo, or equivalent stacks for jobs spanning thousands of accelerators.
- Expertise with FSDP2/HSDP, tensor parallelism, context parallelism, pipeline parallelism, mixed-precision training, and optimization on GPUs, TPUs, or AWS Trainium.
- Pre-training experience and understanding of how model architecture, data representation, batching, and orchestration affect training-system design and efficiency.
- Expertise building large-scale multimodal streaming or hardware-accelerated ingestion with Ray Data, Mosaic Streaming, WebDataset, NVIDIA DALI, NVDEC, or equivalent technologies.
- Deep knowledge of sharding, sampling, packing, caching, prefetching, deterministic data mixing, checkpoint-safe iteration, elastic restart, schemas, dataset versioning, lineage, quality, and reproducible training and evaluation.
- Experience profiling I/O, CPU, memory, network, distributed workers, and accelerators, including diagnosing cross-layer failures and intermittent faults.
Benefits
- US benefits include paid time off, sick time, paid company holidays, paid parental bonding leave, medical, dental and vision insurance, life and disability insurance, and a 401(k) for regular full-time employees.
- Canada benefits include vacation, sick time, parental and adoption leave top-up, extended health, dental and vision coverage, life and disability insurance, and a retirement plan for regular full-time employees.
- Some roles may be eligible for bonus and other incentive programs.
- The posting provides location-specific compensation ranges for British Columbia and California.
Tech Stack
Categories
Data EngineeringML Engineering
About Electronic Arts
Electronic Arts creates next-level entertainment experiences that inspire players and fans around the world. Here, everyone is part of the story. Part of a community that connects across the globe. A team where creativity thrives, new perspectives are invited, and ideas matter. Regardless of your role, team, or location, this is a place where everyone makes play happen. Join us.