1 day ago
Responsibilities
- Architect the on-disk format, indexing structures, transactional metadata layer, Rust data plane, and training data loader.
- Design storage engines and table or file format internals with transactional snapshot isolation, concurrent updates, multimodal schema evolution, global indexing, and low-latency random access.
- Engineer low-level I/O and memory performance through memory mapping, zero-copy layouts, SIMD and vectorized decoding, asynchronous runtimes, and high-throughput object-store clients.
- Build and scale systems for multi-petabyte datasets, millions of files, tens of billions of rows, and large GPU and TPU training clusters.
- Design data-loading systems supporting global shuffling, sequence packing, deterministic checkpoint and resume, and host-to-device memory pipelines.
- Set technical direction across teams, ship production systems at scale, and elevate senior engineers.
- Apply rigorous correctness and verification practices such as differential parity testing, property-based fuzzing, and state-machine modeling for concurrent commits.
Requirements
- 10+ years of experience building storage engines, table or file formats, database internals, or high-performance distributed data and ML systems.
- Proven experience designing and implementing table or file format internals rather than only consuming them through high-level APIs.
- Production systems programming mastery in Rust or C++, with Rust strongly preferred, plus Python and PyO3 for ML framework bindings.
- Deep hands-on expertise in mmap, zero-copy memory layouts, SIMD, vectorized decoding, RoaringBitmaps, tokio, and S3 and GCS object-store client internals.
- Strong understanding of snapshot isolation, MVCC, optimistic concurrency control, manifest trees, and metadata scaling.
- Working knowledge of large-scale distributed training with JAX, Grain, and PyTorch across GPU and TPU meshes.
- Experience setting technical direction across teams, shipping production systems at petabyte scale, and mentoring or elevating senior engineers.
- Bachelor’s, master’s, or doctoral degree in Computer Science or Computer Engineering, or equivalent practical experience.
- Preferred experience contributing to or committing code for Apache Iceberg, Apache Arrow, Parquet, Lance, Vortex, DataFusion, Velox, DuckDB, Polars, Daft, or RocksDB.
- Preferred experience designing ML training data loaders and formats involving PyGrain, ArrayRecord, Riegeli, Megatron-LM, Mosaic StreamingDataset, Ray Data, NVIDIA DALI, or AIStore.
- Preferred experience with multimodal dataset storage and evolving feature or label columns without full dataset rewrites.
- Preferred experience with differential parity testing, property-based fuzzing, or state-machine modeling for concurrent commits.
Categories
Data EngineeringML Engineering
About Apple
Apple designs and sells consumer electronics, software, and services for consumers and professionals worldwide, including iPhone, Mac, iPad, Apple Watch, and AirPods, plus platforms like iOS/macOS and services such as the App Store, iCloud, Music, and TV+. Its business combines device sales with services and subscriptions and in-house silicon design. Founded in 1976, Apple is headquartered in Cupertino, California, and trades on NASDAQ as AAPL.