22 hours ago
London, United KingdomMid Level
Responsibilities
- Construct, calibrate, and execute automated evaluation test suites in Agentflow Studio, measuring task completion, tool-call correctness, latency, and hallucination metrics.
- Curate domain-specific golden datasets, synthetic benchmarks, and regression testbeds with business-domain experts and senior engineers.
- Optimize prompt templates, system instructions, few-shot examples, and structured JSON output parsers using Pydantic and JSON Schema.
- Create and maintain automated test execution and deployment pipelines in Lightspeed CI/CD, enforcing evaluation gates before staging and production rollout.
- Use Python application development expertise to analyze agent errors, evaluate system performance, and recommend coding and prompt improvements.
- Implement security measures and Lakera Guardrail policies to support safe agent functionality.
- Monitor evaluation cycles and provide operational support, performance metrics, and diagnostic reporting to business stakeholders and squad leads.
- Assess risk, follow applicable policies and regulations, and escalate and report control issues transparently.
Requirements
- Relevant software engineering or systems analysis experience with a specialized focus on Python and generative AI application development.
- Strong Python programming proficiency and practical experience integrating REST and gRPC APIs, Docker containers, and Git workflows.
- Working knowledge of AI evaluation frameworks and benchmarking methods including BLEU, ROUGE, semantic similarity, and LLM-as-a-judge.
- Experience implementing automated test pipelines in CI/CD environments such as Lightspeed, Tekton, and Jenkins.
- Ability to manage tight deadlines and unexpected changes in model outputs or business requirements.
- Strong written and verbal communication skills with a methodical, detail-oriented approach to software quality and data integrity.
- Bachelor’s or university degree, or equivalent experience, in Computer Science, Data Science, or a related quantitative discipline.
Benefits
- Hybrid Citi London workplace with up to two days working from home per week.
- Full-time position with a competitive base salary that is reviewed annually.
- 27 days of annual leave plus bank holidays.
- Discretionary annual performance-related bonus.
- Private medical care and life insurance.
- Employee Assistance Program.
- Pension plan.
- Paid parental leave.
- Employee, family, and friend discounts.
About Citi
Citi is a public financial-services company offering consumer and institutional banking, credit cards, wealth management, treasury and trade solutions, and capital-markets services. It serves individuals, corporations, financial institutions, and governments in more than 160 countries and jurisdictions, earning interest and fee income from lending, payments, trading, and advisory. Founded in 1812 and headquartered in New York, it trades on the NYSE under the ticker C.