2 months ago
Remote, PolandMid Level
Responsibilities
- Design and run automated test suites for non-deterministic LLM-generated documentation against source code truth and NLI accuracy thresholds.
- Validate state consistency, schema integrity, and data transformations across Neo4j and Qdrant data pipeline executions.
- Build regression and evaluation frameworks to monitor quality drift, hallucination rates, and pipeline performance across releases.
- Create and maintain golden datasets for benchmarking LLM prompt outputs and retrieval accuracy.
- Collaborate with AI/ML and Data Engineers to define acceptance criteria, defect logging, and pipeline health checks.
Requirements
- At least 3 years of Automation Quality Assurance experience focused on backend systems, complex data pipelines, or AI-driven applications.
- Direct experience testing AI/LLM outputs, including non-deterministic text, confidence scoring, and NLI assertion logic.
- Direct experience with data pipeline and ETL testing, including batch processing, schema integrity, and data transformations.
- Advanced proficiency in Python and test frameworks such as PyTest.
- Hands-on experience querying graph databases, vector search stores, or SQL databases, including Neo4j/Cypher or Qdrant.
- Proficiency with Docker, Git, REST API automation, and integrating tests into CI/CD pipelines.
- Professional English proficiency at B2+/C1 level.
Benefits
- Full remote working possibilities.
- Company-paid premium medical package through Luxmed.
- MyBenefit cafeteria platform with selectable perks.
- Internal meetups, conferences, workshops, Udemy access, language courses, and company-paid certifications.
- Opportunities for internal mobility and work across diverse domains.
- Full-time role in Poland on an approximately 26-week, six-month enterprise initiative.
About Ciklum
Ciklum is an IT services firm that builds custom digital products and provides nearshore engineering teams, QA, DevOps, data analytics, and UX for enterprises and high-growth startups. It operates a services model spanning product engineering, e-commerce solutions, and managed delivery centers. Founded in 2002 and headquartered in London, the privately held company serves global clients including Just Eat and Zurich Insurance.