Member of Technical Staff, Evals & Post-Training Product
Fireworks AI9 months ago
Base Salary
$200k - $350k/yr
Responsibilities
- Design and scale internal evaluation tooling for measuring model quality, comparing model changes, and informing post-training decisions.
- Build and improve user-facing fine-tuning and post-training workflows across SFT, RFT, and related model-improvement capabilities.
- Partner with customers and internal stakeholders to understand needs, support high-priority engagements, triage issues, and turn bespoke workflows into reusable product capabilities.
- Develop APIs, SDKs, backend systems, and web application experiences that help users author evaluations, understand results, fine-tune models, and iterate quickly.
Requirements
- At least 1 year of software engineering experience, with the role hiring across multiple levels.
- Hands-on experience designing LLM evaluations and/or applying post-training methods to guide model improvement.
- Ability to work across backend systems and developer-facing product surfaces, including full-stack feature delivery when needed.
- Understanding of the generative AI lifecycle from prompting and dataset curation through fine-tuning and productionizing agents.
- Willingness to speak with users, triage GitHub issues, and build emerging products from scratch.
- Preferred: 3 or more years of software engineering experience.
- Preferred: experience designing domain-specific evaluations for areas such as medical, legal, coding, or custom internal datasets.
- Preferred: contributions to developer tools or AI/ML open-source repositories.
- Preferred: understanding of GPU constraints, inference optimization, and their relationship to model performance.
- Preferred: experience owning features end to end in fast-paced startup environments.
Benefits
- Opportunity to work on advanced AI infrastructure and model-serving challenges.
- Collaborate with experienced engineers and AI researchers.
- High ownership and direct product impact in a fast-growing startup environment.
- Equal-opportunity employer committed to diversity and inclusion.