Foundation Model Data Engineer

1 month ago

Base Salary

$155k - $210k/yr

Responsibilities

Own the end-to-end strategy and creation of foundation-model pre-training datasets covering web data, code, books, and technical papers.
Design and implement petabyte-scale pipelines for cleaning, exact and fuzzy deduplication, filtering, and signal extraction.
Build post-training datasets including SFT instructions, multi-turn dialogues, RLHF data, DPO data, preference data, human-labeling workflows, and quality gold sets.
Drive acquisition and processing of vision and video data, including multimodal alignment, compression, and temporal consistency.
Develop high-throughput Python ingestion and transformation scripts using multiprocessing, multithreading, and efficient memory management.
Analyze training corpora statistically to identify bias, knowledge gaps, and quality regressions.
Design synthetic-data generation pipelines using existing models for labeling and refinement.

5+ years of industry experience in Data Science or Machine Learning, including a proven record of building and managing datasets for foundation models.
Expert-level Python proficiency focused on high-performance code, multiprocessing, multithreading, and efficient memory management.
Demonstrated experience working with petabyte-scale datasets used to train production-grade LLMs or large vision models.
Experience reconstructing massive LLM training datasets from raw web crawls such as Common Crawl and specialized domain data.
Hands-on experience creating RLHF, DPO, and multi-turn instruction-following datasets, including human-labeling workflows and quality gold sets.
Mastery of data-at-scale frameworks or formats such as Spark, Ray, WebDataset, or Parquet.
Experience building large-scale image or video datasets, multimodal crawling, video processing, codecs, compression, taxonomy design, or data-centric AI is preferred.
A Master’s or PhD in a quantitative field focused on data-centric AI or information retrieval is preferred.

11-50 employees