10 days ago
Kraków, PolandStaff+
Responsibilities
- Own and improve the release-gating LLM-judge pipeline over real and golden conversations.
- Build and maintain evaluation datasets, golden sets, Polish-language coverage, and documented blind spots.
- Perform root-cause analysis on real conversations using a stable issue taxonomy.
- Develop abuse, adversarial, and anomalous-session detection alongside guardrail and red-team work.
- Drive evals-driven development, reproducible release gating, and quality measurement that stakeholders can act on.
- Set and defend technical direction, act as reviewer of record, mentor engineers, scope work with product and engineering management, and present results to stakeholders.
- Make the conversation-mining and evaluation platform diagnostic, reproducible, and extensible.
Requirements
- 5+ years building and running production software.
- 2+ years working on LLM-based systems used by real users.
- Strong production engineering fundamentals, including testing, CI/CD, containers, observability, and cloud experience.
- Demonstrable experience evaluating generative systems, including LLM-as-judge, human-label calibration, inter-annotator agreement, regression suites, and offline-versus-online divergence.
- Experience with prompt and context engineering, structured outputs, failure-mode taxonomies, and production agentic primitives such as tool use, multi-turn state, and MCP.
- Experience with LLM evaluation and observability tooling such as LangFuse, Braintrust, or Weave, or equivalent systems built independently.
- Comfort working with SQL, data lakes or warehouses, and reproducible metrics.
- Fluency with AI-assisted development tools such as Claude Code, Cursor, and Copilot.
- Ability to explain technical arguments clearly to non-technical audiences.
- English at B2 level and Polish-language fluency sufficient to read and judge user conversations.
- Nice-to-have experience with harness and loop engineering, auto-improving systems, adversarial robustness, jailbreak testing, red-teaming, abuse or fraud detection, and e-commerce, search, or recommendation domains.
Benefits
- Hybrid working from Warsaw or Kraków.
- B2B type of contract.
- Flexible hours and an excellent working environment.
- Access to a released product serving millions of users, frontier models across multiple providers, and an open-source track.
- Ownership of a greenfield-in-practice platform with executive attention and opportunities to shape a growing team.
- Benefits and training opportunities for professional development.
- Opportunity to contribute to InPost’s sustainability, convenient delivery, and circular-economy mission.
Categories
