2 months ago
Remote, IndiaSenior
Responsibilities
- Design, build, and maintain automated frontend, backend, API, contract, workflow, security, performance, and reliability test suites.
- Test Temporal workflows through worker kills, provider outages, retry storms, idempotency checks, and compensation or saga verification.
- Verify multi-tenant isolation, data access controls, configuration boundaries, cost attribution, sandboxing, egress controls, and disallowed tool-call behavior.
- Develop LLM-specific tests using statistical assertions, semantic-similarity scoring, confidence thresholds, flakiness quarantine, adversarial corpora, and prompt-injection probes.
- Validate Human-in-the-Loop escalation, golden datasets, synthetic-data pipelines, evaluation infrastructure, and CI integration.
- Automate performance and NFR verification for invocation overhead, concurrency, queue backpressure, provider failover, tracing, token accounting, and alerting.
- Integrate test suites into CI/CD as release gates and produce reproducible evidence packs for client milestone sign-offs.
- Maintain test environments, mocked LLM/provider layers, synthetic-data generators, and reusable test frameworks.
- Mentor mid-level QA engineers and contribute AI testing patterns to the internal quality engineering practice and AI Academy.
Requirements
- 8+ years of test automation experience, including API testing and building reusable frameworks adopted by other engineers.
- Hands-on experience testing LLM-powered or other non-deterministic systems using statistical assertions, semantic scoring, and model-variance analysis.
- Experience testing asynchronous or event-driven systems with failure injection, idempotency verification, and eventual-consistency assertions; Temporal or a similar workflow engine is preferred.
- Strong CI/CD integration skills, including GitHub Actions or equivalent, and experience designing release gates.
- Solid SQL and data-validation skills, including synthetic test-data pipelines and dataset-quality measurement.
- Security-testing and adversarial-testing experience, including prompt-injection techniques or a strong AppSec testing background.
- Strong proficiency in Playwright and TypeScript is mandatory.
- Deep understanding of LLM behavior, prompt sensitivity, and common RAG failure modes.
- Strong statistical-analysis skills for interpreting evaluation scores and advising AI engineers on prompt tuning.
- Experience with performance or load tools such as k6 or Locust is desirable.
- Experience with multi-tenant SaaS isolation, evaluation and observability stacks, compliance evidence, accessibility testing, document extraction, LLM-generated test assets, high-concurrency background processing, or qTest is desirable.
- Commercial Real Estate workflow or lease-accounting experience is desirable.
Benefits
- Friendly, open-door community and collaboration with experienced professionals.
- Large-scale projects, global impact, internal mobility, and opportunities to expand expertise.
- Internal meetups, conferences, workshops, Udemy access, language courses, and company-paid certifications.
- Company-paid medical insurance, mental health support, and financial and legal consultations.
- Full-time position in India with teams in Chennai and Pune.
Tech Stack
Categories
About Ciklum
Ciklum is an IT services firm that builds custom digital products and provides nearshore engineering teams, QA, DevOps, data analytics, and UX for enterprises and high-growth startups. It operates a services model spanning product engineering, e-commerce solutions, and managed delivery centers. Founded in 2002 and headquartered in London, the privately held company serves global clients including Just Eat and Zurich Insurance.