9 hours ago
Kraków, PolandSenior
Responsibilities
- Own and improve the LLM-judge pipeline used to gate releases over real and golden conversations.
- Build and maintain evaluation datasets, golden sets, coverage documentation, and Polish-language test coverage.
- Perform root-cause analysis on production conversations and maintain a stable issue taxonomy with comparable trends.
- Develop detection for abusive, adversarial, anomalous, and suspicious sessions alongside guardrail and red-team work.
- Drive evals-driven development by defining measurable quality criteria before features are built.
- Build agent loops, retries, fallbacks, tool-call orchestration, deterministic replay, and other infrastructure that makes non-deterministic systems testable.
- Connect production signals to prompt, retrieval, and routing improvements and automate failure-mining workflows.
- Set technical direction, act as reviewer of record, mentor engineers, scope work with the PM and EM, and present results to stakeholders.
Requirements
- 5+ years of experience building and running production software, including 2+ years working on LLM-based systems used by real users.
- Strong production engineering fundamentals, including testing, CI/CD, containers, observability, and cloud-based development.
- Demonstrable experience evaluating generative systems through LLM-as-judge methods, human-label calibration, inter-annotator agreement, regression suites, and offline-versus-online divergence analysis.
- Experience with prompt and context engineering, structured outputs, context-window budgeting, failure-mode taxonomies, and production agentic primitives such as tool use, multi-turn state, MCP, and agent-to-agent integration.
- Experience with LLM evaluation and observability tooling such as LangFuse, Braintrust, Weave, or equivalent tools built independently.
- Comfort working with data at scale, including SQL, data lakes or warehouses, and reproducible metrics.
- Fluency with AI-assisted development tools including Claude Code, Cursor, and Copilot.
- Ability to explain technical arguments clearly to non-technical audiences.
- English at B2 level and Polish-language proficiency sufficient to read user conversations.
- Preferred experience with harness and loop engineering, auto-improving systems, adversarial robustness, jailbreak testing, red-teaming, abuse or fraud detection, and e-commerce, search, or recommendation domains.
Benefits
- Hybrid working from Warsaw or Kraków.
- B2B type of contract.
- Flexible hours and an excellent working environment.
- Range of benefits and training opportunities for professional development.
- Opportunity to own a greenfield-in-practice platform serving a production product used by millions of customers.
- Direct access to frontier models at committed capacity across multiple providers and an open-source track.
- Opportunity to shape an expanding team and contribute to sustainability, convenient deliveries, and the circular economy.
Categories
About InPost
InPost is a public logistics company listed on Euronext Amsterdam that builds and operates an out-of-home parcel delivery network for e-commerce. Its network includes more than 50,000 out-of-home pick-up points, including over 27,000 parcel lockers, plus to-door courier and fulfillment services for online merchants across Poland, the UK, Italy, Iberia, France and Benelux. Headquartered in Luxembourg, it acquired French operator Mondial Relay in 2021.