5 days ago
Base Salary
$121k - $182k/yr
Responsibilities
- Construct, calibrate, and execute automated evaluation test suites in Agentflow Studio.
- Curate domain-specific golden datasets, synthetic benchmarks, and regression testbeds with domain experts and senior engineers.
- Optimize prompt templates, system instructions, few-shot examples, and Pydantic/JSON Schema output parsers.
- Author and maintain automated test execution and deployment pipelines in Lightspeed, enforcing evaluation gates before staging and production rollout.
- Analyze complex agent errors, evaluate system performance, and recommend coding and prompt solutions using Python.
- Implement security measures and Lakera Guardrail policies to support safe agent functionality.
- Monitor evaluation cycles and provide operational support, performance metrics, and prompt diagnostic reporting to stakeholders and squad leads.
- Assess risk, follow applicable policies and regulations, and escalate and report control issues transparently.
Requirements
- Bachelor’s or university degree in Computer Science, Data Science, or a related quantitative discipline, or equivalent experience.
- 5–8 years of relevant software engineering or systems analysis experience, or 2–4 years with specialized Python and generative AI application development experience.
- Strong Python programming proficiency and practical experience integrating REST/gRPC APIs, Docker containers, and Git workflows.
- Working knowledge of AI evaluation frameworks and benchmarking methods including BLEU, ROUGE, semantic similarity, and LLM-as-a-judge.
- Experience managing and implementing automated test pipelines in Lightspeed, Tekton, or Jenkins.
- Ability to manage tight deadlines and changing model outputs or business-line requirements.
- Strong written and verbal communication skills with a methodical, detail-oriented approach to software quality and data integrity.
Benefits
- Medical, dental, and vision coverage.
- 401(k), life, accident, and disability insurance.
- Wellness programs.
- Paid vacation, sick leave, and holidays.
- Offerings may vary by jurisdiction, job level, and date of hire.
About Citi
Citi is a public financial-services company offering consumer and institutional banking, credit cards, wealth management, treasury and trade solutions, and capital-markets services. It serves individuals, corporations, financial institutions, and governments in more than 160 countries and jurisdictions, earning interest and fee income from lending, payments, trading, and advisory. Founded in 1812 and headquartered in New York, it trades on the NYSE under the ticker C.