Responsibilities
- Design and run automated test suites for non-deterministic LLM-generated documentation against source code truth and NLI accuracy thresholds.
- Validate state consistency, schema integrity, and data transformations across Neo4j and Qdrant data pipeline executions.
- Build regression and evaluation frameworks to monitor quality drift, hallucination rates, and pipeline performance across releases.
- Create and maintain golden datasets for benchmarking LLM prompt outputs and retrieval accuracy.
- Collaborate with AI/ML and Data Engineers to define acceptance criteria, defect logging, and pipeline health checks.
Requirements
- At least 3 years of Automation Quality Assurance experience focused on backend systems, complex data pipelines, or AI-driven applications.
- Direct experience testing AI/LLM outputs, including non-deterministic text, confidence scoring, and NLI assertion logic.
- Direct experience with data pipeline and ETL testing, including batch processing, schema integrity, and data transformations.
- Advanced proficiency in Python and test frameworks such as PyTest.
- Hands-on experience querying graph databases, vector search stores, or SQL databases, including Neo4j/Cypher or Qdrant.
- Proficiency with Docker, Git, REST API automation, and integrating tests into CI/CD pipelines.
- Professional English proficiency at B2+/C1 level.
Benefits
- Full remote working possibilities.
- Company-paid premium medical package through Luxmed.
- MyBenefit cafeteria platform with selectable perks.
- Internal meetups, conferences, workshops, Udemy access, language courses, and company-paid certifications.
- Opportunities for internal mobility and work across diverse domains.
- Full-time role in Poland on an approximately 26-week, six-month enterprise initiative.
About Ciklum
Ciklum is a global Experience Engineering firm that stands at the forefront of innovation, blending next-generation product engineering, exceptional customer experiences, and cutting-edge AI. The business revolutionizes the way people live by developing technologies that reimagine, reshape, and redefine the future. For over 20 years, Ciklum has been a trusted partner to global enterprises and digital disruptors, spearheading the creation of digital solutions that tackle complex corporate challenges while propelling businesses toward accelerated growth and success. Its global team of over 4,000 highly skilled engineers, experience design specialists and consultants joins forces to engineer technology that paves the way for the future of your business. Ciklum builds tailored digital solutions that leverage emerging technologies for such clients as Just Eat, Metro Markets, EFG International, Zurich Insurance and others. For more information about us visit www.ciklum.com