21 hours ago
Kraków, PolandSenior
Responsibilities
- Define and implement evaluation strategies for LLM and AI applications used in Credit and Lending.
- Build and maintain evaluation datasets and repeatable test harnesses for prompts, models, retrieval-augmented generation, and agentic workflows.
- Measure response correctness, relevance, groundedness, completeness, latency, and cost through automated and human evaluation.
- Evaluate retrieval quality, source attribution, and answer grounding across financial data and vector databases.
- Test AI systems for robustness, hallucinations, bias, fairness, prompt injection, and other safety risks.
- Automate evaluation and regression tests in CI/CD before prompt, model, or data changes reach production.
- Analyze evaluation results, investigate quality regressions, and communicate trade-offs and recommendations.
- Collaborate with engineers, product managers, and business experts to align evaluation coverage with user and business needs.
Requirements
- Deep expertise in Python and its ML and AI ecosystem.
- Experience with FastAPI, LangChain, and data containerisation.
- Understanding of microservices, observability, API design, and concurrency models.
- Experience building and deploying on Azure or GCP using containers, Kubernetes, and CI/CD tools such as Jenkins, Azure DevOps, or GCP Cloud Build.
- Practical experience with Test-Driven Development, Behaviour-Driven Development, and automated testing.
- Strong understanding of LLM architectures, including fine-tuning models on domain-specific data.
- Hands-on experience with Retrieval-Augmented Generation, advanced prompt engineering, and agentic architectures.
- Clear English communication skills, including explaining technical solutions, evaluation results, and trade-offs to business partners.
Benefits
- Monthly gross pay range of 17484 PLN–26225 PLN, with discretionary variable pay influenced by group, business/function, and individual performance.
- Healthcare, family-friendly leaves, pension, and life assurance benefits.
- Recognition awards, a Multisport card, private medical care, and life insurance.
- One-time reimbursement of home-office setup costs up to 800 PLN.
- Cafeteria platform, employee assistance program, additional PPK contributions, corporate events, CSR initiatives, nursery discounts, training and education support, social fund, flexible working hours, and free parking.
- Recruitment includes an online behavioural test and telephone screen for external candidates, followed by a hiring manager interview.
Tech Stack
Categories
About HSBC
HSBC is a global bank that provides retail, commercial and investment banking, payments, and wealth management to individuals, SMEs, and multinationals. It earns fees, interest income, and trading revenues through branches and digital platforms across major markets. Headquartered in London, it serves over 40 million customers in 58 countries and is listed in London, Hong Kong, New York, and Bermuda.