Director, AI Assisted Solutions-1
College Board4 days ago
Remote, United StatesStaff+
Base Salary
$88k - $135k/yr
Responsibilities
- Lead end-to-end delivery of AI solution engagements for College Board assessment programs, from problem definition and requirements through tool development, quality sign-off, training, and adoption.
- Design and build AI tools, prompts, workflows, automated item-generation solutions, and agentic or multi-agent processes for assessment content development.
- Create rubrics, benchmarks, scoring frameworks, and other systematic evaluation criteria for AI-generated assessment content.
- Evaluate AI outputs for construct alignment, flawed reasoning, bias, factual errors, and other quality issues, refining prompts, context, and workflows as needed.
- Identify scaling requirements and gaps across culture, staffing, processes, technology, and program roadmaps.
- Contribute assessment expertise and engagement evidence to enterprise-wide agentic workflow standards, hybrid human-agent architecture, shared tooling, and strategy.
- Design and deliver training, process documentation, adoption plans, and impact metrics for content teams.
- Experiment with emerging AI tools, models, and techniques, share findings internally, and represent College Board at relevant external meetings and conferences.
Requirements
- 7+ years of standardized assessment item-development experience across multiple item types, including multiple-choice, free-response, discrete, and set-based items.
- 3+ years of hands-on professional experience applying automated item generation, retrieval-augmented generation, prompt engineering, context engineering, or related AI-assisted and agentic techniques.
- Expertise in assessment for Reading Comprehension, Writing, Math, or Science, including advanced item-writing craft and construct alignment.
- Experience bringing process or technology improvements into adoption with measurable effects on content-production efficiency or quality.
- Working knowledge of large language model capabilities and limitations, including hallucination, context-window constraints, and training-data staleness.
- Experience with agentic or multi-agent workflows, including generator/reviewer or generator/critic patterns, tool-calling, and chained or orchestrated prompts.
- Experience with vibe coding or coding and rapid prototyping, including independently building scripts, tools, or interactive prototypes.
- Experience designing processes, educating others, enabling adoption, and working with item-development lifecycles, authoring and review tools, and content-management or item-banking systems.
- Ability to translate between technical and non-technical stakeholders, manage strategic initiatives and project plans, maintain documentation, and collaborate across functions.
- Ability to work effectively amid ambiguity, maintain strong organization and attention to detail, and travel 6–8 times per year.
- A master’s degree or equivalent education and experience is strongly preferred.
- Authorization to work in the United States is required.
Benefits
- Fully remote full-time position; candidates near College Board offices may choose fully remote work or hybrid work on Tuesdays and Wednesdays.
- Competitive compensation adjusted by location and based on qualifications, experience, impact, and market data.
- Meaningful mission-driven work at a self-sustaining nonprofit with a supportive team and comprehensive benefits package.
- Travel to relevant professional meetings and conferences approximately 6–8 times per year.
Categories
About College Board
College Board is a nonprofit education organization that builds and administers programs such as the SAT, PSAT/NMSQT, Advanced Placement, CLEP, and BigFuture planning tools for students, schools, and colleges. It generates revenue primarily through exam and program fees and services to educational institutions. Founded in 1900 and headquartered in New York, it works with U.S. secondary schools and higher-ed admissions to deliver assessments and course frameworks.