7 months ago
Base Salary
$214k - $301k/yr
Responsibilities
- Train, fine-tune, evaluate, and diagnose biological foundation models with research partners.
- Evaluate and curate training and validation datasets at petabyte scale.
- Design libraries and abstractions for researchers and engineers.
- Optimize GPU-native data loading and resolve utilization bottlenecks across hardware and software backends.
- Build and maintain reliable distributed pre-training and post-training systems.
- Support production deployment and track model and dataset artifacts throughout the ML lifecycle.
- Improve model and cluster monitoring, unit and integration testing, and continuous integration.
- Deploy models and supporting infrastructure in external environments with partner researchers and engineers.
Requirements
- 5+ years of experience building ML systems or distributed infrastructure at scale.
- Strong Python, PyTorch, and CUDA skills, including custom training loops and knowledge of model architectures, optimization, and evaluation.
- Experience with large-scale distributed systems, multi-GPU training, checkpointing, and failure recovery.
- Ability to diagnose performance issues across GPUs, memory, networking, storage, and data loading.
- Experience building large-scale data pipelines and shared research tools, with familiarity with Zarr or HDF5.
- Experience deploying and monitoring models, tracking model and dataset versions, and maintaining automated tests and continuous integration.
- Familiarity with containers and orchestration tools such as Slurm, Docker, and Kubernetes.
Benefits
- Generous employer match on 401(k) contributions.
- Paid time off to volunteer at an organization of choice.
- Funding for select family-forming benefits.
- Relocation support for employees who need assistance moving.
- The posting references a hybrid work arrangement and potential locations in Redwood City, California, and New York City, New York.
Tech Stack
Categories
About Biohub
Biohub is a nonprofit research organization in Redwood City that builds AI-enabled biology platforms, high-throughput imaging, and lab capabilities to advance disease research for the global scientific community. Funded by the Chan Zuckerberg Initiative as part of the Chan Zuckerberg Biohub Network, it collaborates with leading universities and operates shared facilities rather than selling commercial products. Programs span cell atlases, genomics, and infectious disease, and include an aquatic model-organism facility supporting imaging-focused studies.