2 months ago
San Francisco, CA, USAMid Level
Base Salary
$155k - $200k/yr
Responsibilities
- Integrate internal and open-weight language and multimodal models into GPU evaluation and inference environments.
- Build automated benchmarks for model quality, latency, throughput, memory usage, and systems performance.
- Create reproducible comparisons across company models, external baselines, and runtime configurations.
- Build and maintain experiment tracking, model registries, and versioning for models, datasets, and evaluation configurations.
- Automate workflows from research checkpoints to validated deployments.
- Monitor model quality and systems performance and diagnose pipeline failures or regressions.
- Develop reusable tools for researchers to launch evaluations, compare experiments, and reproduce results.
- Profile end-to-end model workloads and collaborate with distributed systems, inference, and GPU kernel engineers.
Requirements
- At least 2 years of professional ML or software engineering experience, including production ML systems, ML platforms, or MLOps infrastructure.
- Strong Python and software engineering skills with experience building reliable production systems.
- Hands-on experience developing, running, or evaluating Transformer-based models such as BERT, ViT, VLMs, or LLMs using PyTorch, TensorFlow, or JAX.
- Experience implementing model evaluation or benchmarking pipelines, including metrics, reproducibility, and model comparisons.
- Basic understanding of LLM evaluation metrics and familiarity with experiment tracking and model lifecycle workflows.
- Experience running, benchmarking, and debugging models with GPU inference runtimes such as vLLM, SGLang, or TensorRT-LLM in containerized cloud or on-premises environments.
- MS or PhD in Computer Science, Computer Engineering, Machine Learning, or a related technical field, or equivalent practical experience.
- Clear technical documentation and effective collaboration across research, infrastructure, and product engineering teams.
- Familiarity with Hugging Face Transformers or similar model libraries is preferred.
- Experience enabling models on AMD GPUs and ROCm is preferred.
- Open-source contributions to evaluation, model, or ML infrastructure projects are preferred.
Benefits
- Medical, dental, and vision insurance.
- 401k plan.
- Daily lunch, snacks, and beverages.
- Flexible time off.
- Competitive salary and equity.
Categories
