6 months ago
San Francisco, CA, USA or New York, NY, USAMid Level
H1B Sponsor
Base Salary
$98k - $140k/yr
Responsibilities
- Design, test, and iterate on system prompts, tool prompts, and context strategies.
- Analyze production transcripts, logs, and user feedback to reproduce issues and identify root causes.
- Design evaluation strategies, build datasets, run evaluations, and track quality over time.
- Evaluate and launch models from OpenAI, Anthropic, Google, and other research labs across quality, latency, cost, and edge cases.
- Define quality goals and priorities, communicate issue severity and frequency, and drive fixes with engineering and product teams.
- Help manage AI observability and evaluation platforms such as Braintrust.
- Create playbooks and tools that enable teams across Notion to build AI products.
Requirements
- Experience with LLMs, prompting, or AI products.
- Ability to derive insights from large datasets using SQL, coding agents, or other tools.
- Strong analytical instincts and ability to identify signal in noisy data.
- Clear communication skills for explaining complex issues simply.
- Curiosity about LLM capabilities and how AI products perform in real-world use.
- A proactive, high-ownership approach to ambiguous and fast-moving problems.
- Background in engineering, product, data science, research, or consulting is preferred.
- Experience building an independent side project, startup, or tool is preferred.
Benefits
- In-person collaboration is required on Mondays, Tuesdays, and Thursdays, with additional office days possible for certain teams or positions.
- Competitive cash compensation, equity, and benefits are offered.
- The role is based in an office setting, with listed locations including San Francisco and New York City.
Tech Stack
Categories
About Notion
Notion blends your everyday work tools into one. Product roadmap? Company wiki? Meeting notes? With Notion, they're all in one place, and totally customizable to meet the needs of any workflow. It's the all-in-one workspace for you, your team, and your whole company. We humans are toolmakers by nature, but most of us can't build or modify the software we use every day — arguably our most powerful tool. Our team at Notion is on a mission to make it possible for everyone to shape the tools that shape their lives.