7 months ago
Redwood City, CA, USASenior / Staff+
H1B sponsor
Base Salary
$214k - $301k/yr
Responsibilities
- Train, fine-tune, evaluate, and diagnose biological foundation models with researchers.
- Evaluate and curate petabyte-scale training and validation datasets.
- Design libraries and abstractions for researchers and engineers.
- Optimize GPU-native data loading and resolve hardware utilization bottlenecks across backends.
- Build and maintain reliable distributed pre-training and post-training systems.
- Support production model deployment and track model and dataset artifacts throughout the ML lifecycle.
- Improve model and cluster monitoring, unit and integration testing, and continuous integration.
- Deploy models and supporting infrastructure in external environments with partner researchers and engineers.
Requirements
- 5+ years of experience building ML systems or distributed infrastructure at scale.
- Strong Python, PyTorch, and CUDA skills, including custom training loops and knowledge of model architectures, optimization, and evaluation.
- Experience with large-scale distributed systems, multi-GPU training, checkpointing, and failure recovery.
- Ability to diagnose performance issues across GPUs, memory, networking, storage, and data loading.
- Experience building large-scale data pipelines and shared research tools.
- Familiarity with scientific data formats such as Zarr or HDF5.
- Experience deploying and monitoring models, tracking model and dataset versions, and maintaining automated tests and continuous integration.
- Familiarity with containers and orchestration tools such as Slurm, Docker, and Kubernetes.
Benefits
- Generous employer match on employee 401(k) contributions.
- Paid time off to volunteer at an organization of choice.
- Funding for select family-forming benefits.
- Relocation support for employees who need assistance moving.
- The anticipated role locations are Redwood City, California, and New York City, New York, with a hybrid arrangement indicated by the listing.
Tech Stack
Categories
About Biohub
Biohub is a nonprofit research organization in Redwood City that builds AI-enabled biology platforms, high-throughput imaging, and lab capabilities to advance disease research for the global scientific community. Funded by the Chan Zuckerberg Initiative as part of the Chan Zuckerberg Biohub Network, it collaborates with leading universities and operates shared facilities rather than selling commercial products. Programs span cell atlases, genomics, and infectious disease, and include an aquatic model-organism facility supporting imaging-focused studies.