AI Platform Senior Engineer, QA
Accrete AI1 month ago
Mumbai, IndiaSenior
Responsibilities
- Test multi-agent workflows, orchestration layers, cross-domain intelligence systems, conversational AI, and chat-based agents.
- Validate AI-generated narratives, themes, analytical outputs, groundedness, coherence, hallucinations, stability, consistency, and output drift.
- Design and maintain automation frameworks and reusable test utilities using Python, JavaScript/TypeScript, Pytest, and Playwright.
- Build end-to-end automation for user interfaces, APIs, and agent pipelines and integrate automation with CI/CD pipelines.
- Validate AI/ML models and inference pipelines, including LLM-as-Judge, semantic similarity, statistical, and comparative testing.
- Perform API, load, stress, performance, and scalability testing and identify bottlenecks and risks.
- Analyze requirements and AI specifications, design test strategies, track defects and quality metrics, and assess release readiness.
- Collaborate with Engineering, Product, and Data Science teams to improve QA processes.
Requirements
- 6–8+ years of experience in QA, automation, or quality engineering.
- Strong proficiency with Python, JavaScript/TypeScript, Pytest, and Playwright.
- Experience testing AI systems, chatbots, or LLM-based platforms and non-deterministic system behavior.
- Experience with API testing and hands-on performance testing.
- Strong analytical, debugging, communication, and documentation skills.
- Preferred: experience with LangChain, AutoGen, CrewAI, RAG systems, vector databases, MLOps, AI observability tools, AWS, statistical validation, NLP evaluation, GitHub Actions, Jenkins, GitLab, Docker, and Kubernetes.
- Preferred: experience working with CI/CD and familiarity with agent frameworks and cloud platforms.
Benefits
- ₹10 lakh Mediclaim coverage for the employee and family.
- Flexible PTO and hybrid work with remote flexibility according to company guidelines.
- Professional learning opportunities and career advancement support.
- Catered lunches and snacks.
- Company-sponsored social events and team activities.
Tech Stack
Categories
About Accrete AI
Accrete AI builds an AI agent platform and managed service that automates judgment-heavy analysis and workflows for government and enterprise customers. Its core Knowledge Engine unifies siloed data into an auditable context graph that expert agents use to support high-stakes decisions across national security and commercial operations. Founded in 2017 and headquartered in New York City, the privately held company offers subscription software and services engagements.