Member of Technical Staff, Evals & Post-Training Product
Fireworks AI1 year ago
San Mateo, CA, USAEntry Level / Mid Level / Senior
H1B sponsor
Base Salary
$200k - $350k/yr
Responsibilities
- Design and scale internal evaluation tooling for measuring model quality, comparing model changes, and informing post-training decisions.
- Build and improve user-facing fine-tuning and post-training workflows, including SFT and RFT experiences.
- Develop APIs, SDKs, backend systems, and web application surfaces that help users author evals, understand results, fine-tune models, and iterate quickly.
- Partner with customers and internal stakeholders to understand needs, support high-priority engagements, and triage issues.
- Convert bespoke evaluation and fine-tuning workflows into reusable product capabilities.
Requirements
- 1–7 years of software engineering experience, with the company hiring at multiple levels.
- Hands-on experience designing LLM evaluations and/or using post-training methods to guide model improvement.
- Ability to work across backend systems and developer-facing product surfaces, including shipping full-stack features when needed.
- Understanding of the GenAI lifecycle from prompting and dataset curation through fine-tuning and productionizing agents.
- Willingness to speak with users, triage GitHub issues, and build products from scratch.
- Preferred: 3+ years of software engineering experience.
- Preferred: experience designing and running domain-specific evaluations for areas such as medical, legal, coding, or custom datasets.
- Preferred: contributions to developer tools or AI/ML repositories.
- Preferred: understanding of GPU constraints, inference optimization, and their relationship to model performance.
- Preferred: experience owning features end-to-end in fast-paced environments.
Tech Stack
Categories
About Fireworks AI
Fireworks AI builds a generative AI platform for developers and enterprises to train, fine-tune, and serve open models for production use across text, image, audio, embeddings, and multimodal workloads. It offers managed inference and tooling via APIs on globally distributed infrastructure, with a usage-based SaaS model. Founded in 2022 and headquartered in San Mateo, CA, Fireworks AI is a privately held, Series D company.