Member of Technical Staff - Large Scale Data Infrastructure
Black Forest Labs3 months ago
Base Salary
$180k - $300k/yr
Responsibilities
- Build scalable data loaders for training runs across thousands of GPUs.
- Create efficient storage and retrieval systems for petabyte-scale datasets.
- Implement multi-cloud object storage abstractions.
- Execute large-scale data migrations across various storage systems.
- Debug and resolve performance bottlenecks in distributed data loading.
Requirements
- Experience building and operating data pipelines at petabyte scale.
- Proficiency in optimizing data loading processes.
- Familiarity with petabyte-scale video and image datasets.
- Ability to write processing jobs that operate on millions of files.
- Experience debugging distributed system bottlenecks across large fleets of machines.
- Nice to have: experience with streaming dataset formats and video codec internals.
Benefits
- Flexible work arrangements with options for remote work and in-person collaboration.
- Coverage of reasonable travel costs for in-person meetings.
- A collaborative team culture grounded in values of obsession, low ego, boldness, and kindness.