1 day ago
Remote, United StatesSenior
Base Salary
$105k - $165k/yr
Responsibilities
- Own system prompts, tool descriptions, and evaluation prompts for Personalized Credit Insights and the AI chat assistant.
- Build evaluation frameworks, test cases, regression suites, and trusted reporting from production failures and evidence.
- Measure LLM evaluator reliability against human-labeled data using precision, recall, and agreement metrics.
- Design and monitor guardrails including classifiers, PII redaction, prompt-injection defenses, and financial-advice compliance rules.
- Lead model upgrades by comparing quality, cost, and latency and shipping changes safely through A/B tests.
- Reduce cost and latency through prompt caching, model routing, and context management.
- Build transcript viewers, labeling workflows, dashboards, and other tools for Conversation Analysts and QA engineers.
- Partner with Analytics, Legal, and Compliance on fairness and bias evaluation of AI outputs.
- Use AI coding assistants and agentic workflows while reviewing and refining generated output.
Requirements
- Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
- Demonstrated professional software engineering experience with strong Python skills and the ability to read and contribute to Java Spring Boot microservices.
- Proven experience building and operating production LLM-powered applications, including prompt design, tool calling, structured outputs, and streaming.
- Experience building LLM evaluations including test sets, regression suites, and LLM-as-judge approaches, with evaluator quality measured against human labels.
- Working knowledge of Amazon Bedrock or a comparable managed LLM platform and AWS services including S3, Lambda, and DynamoDB.
- Proficiency in SQL and comfort analyzing production data in a warehouse such as Amazon Redshift.
- Production ownership experience including code review, release management, and on-call support within Agile/Scrum teams.
- Strong written and verbal communication skills for explaining evaluation results and tradeoffs to product, legal, and executive stakeholders.
- Experience with guardrail tooling, LLM security testing, RAG and retrieval evaluation, fairness and bias evaluation, model migrations, Datadog, Tableau, or regulated domains such as FCRA, ECOA, UDAAP, and CFPB guidance is a plus.
Benefits
- Inclusive culture reflecting FICO’s core values and an opportunity for professional development and learning.
- Competitive compensation, benefits, and rewards programs.
- People-first work environment with work/life balance, employee resource groups, and social events.
Categories
About FICO
FICO (Fair Isaac Corporation) builds enterprise decision-management software, analytics, and the FICO Score used by lenders to assess consumer credit risk. It sells B2B software, cloud services, and scoring products to banks, insurers, telecoms, and other regulated industries, plus consulting. Founded in 1956 and headquartered in Bozeman, Montana, FICO is a public company on the NYSE; the FICO Score is used by most top U.S. lenders and available in 40+ countries.