1 day ago
Sunnyvale, CA, USAMid Level
H1B sponsor
Base Salary
$110k - $170k/yr
Responsibilities
- Manage and maintain technical interview questions, metadata, variants, validation checks, and difficulty calculations.
- Design scoring workflows that compare candidate answers with reference answers and rubrics.
- Calibrate and benchmark LLM prompts and scoring models against human-graded samples.
- Build end-to-end automation for question selection, test delivery, answer collection, scoring, and reporting.
- Monitor scoring consistency, accuracy, bias, and pipeline health while routing low-confidence cases for human review.
- Document prompts, scoring logic, and processes for auditability.
- Help transition source code and repositories to an internal Git platform and support broader internal automation initiatives.
Requirements
- Bachelor's degree in Computer Science or a related field, or equivalent practical experience.
- Solid programming fundamentals in Python, JavaScript/TypeScript, or a similar language.
- Experience working with APIs, including LLM and AI APIs such as OpenAI and Anthropic.
- Demonstrated experience designing and evaluating prompts or pipelines for LLM-based tools.
- Proficiency with Git and a hosted platform such as GitHub or GitLab, including branching, pull or merge requests, code review, and CI basics.
- Basic understanding of databases and SQL or similar technologies.
- Strong attention to detail, technical writing ability, and the ability to work independently in an evolving role.
- Preferred qualifications include assessment-platform or ATS integration experience, scripting or workflow automation exposure, interest or coursework in ML fairness and bias, and technical interview experience.
Benefits
- Medical, dental, vision, life, and disability insurance.
- 401(k), 11 paid holidays, vacation time, sick time, and a comprehensive leave program.
- Eligibility to participate in the Fortinet equity program and potential bonus eligibility.
Tech Stack
Categories
About Fortinet
Fortinet is a public cybersecurity company that builds network security hardware and software for enterprises, service providers, and governments. Its FortiGate firewalls, secure networking (SD-WAN), endpoint and cloud security, and threat intelligence subscriptions are sold via direct and channel partners. Founded in 2000 and headquartered in Sunnyvale, California, Fortinet trades on NASDAQ (FTNT) and serves more than 660,000 customers worldwide.