3 hours ago
Bengaluru, IndiaMid Level
H1B Sponsor
Responsibilities
- Design and implement comprehensive QA strategies for AI/ML systems, including accuracy, bias, robustness, regression, and integration testing.
- Create self-contained evaluation tasks with prompts, supporting files, and detailed grading rubrics.
- Automate validation suites for agentic and multi-agent systems, ML models, integration testing, and CI/CD pipelines.
- Validate that AI/ML models consume accurate, authorized, properly structured, and high-quality data during training and inference.
- Observe and document AI agent behavior, model performance, hallucinations, and data-grounded output accuracy.
- Refine evaluation tasks and rubrics using feedback and collaboration to improve benchmarking methodologies.
- Ensure AI data sources and structures meet security, privacy, governance, regulatory, and compliance standards.
- Collaborate across IT, engineering, operations, data, integrations, PMO, CRM, risk, compliance, and business technology teams.
- Monitor developments in AI and provide technical recommendations to the organization.
Requirements
- Proficiency in Python, TypeScript, or other programming languages used in AI and test automation.
- Experience designing or applying rubric-based evaluations, structured grading, or scoring frameworks.
- Direct experience with LLM evaluation frameworks and benchmarking tools such as LangSmith and Confident AI.
- Knowledge of the GenAI stack, including Retrieval Augmented Generation (RAG).
- Experience with LLM solutions such as Azure OpenAI, Vertex AI, or ChatGPT Enterprise.
- Experience with cloud platforms including Azure, GCP, or AWS.
- Thorough understanding of DevOps, automation, and CI/CD tools including GitHub and Terraform.
- Understanding of AI ethics, risk management, and data governance.
- Strong attention to detail and ability to identify subtle data hallucinations, logic errors, and inconsistencies.
- Advanced written communication skills for documenting nuanced observations and feedback.
Benefits
- Competitive pay based on the work performed rather than previous salary.
- Health coverage for employees and families in many locations.
- Distributed work environment with flexible locations and schedules for many roles.
- Generous vacation allowance.
- Employer matching of up to $2000, or local currency equivalent, for financial donations and service.
- Up to 40 hours per year for volunteer projects.
- At least 16 weeks of parental leave.
Categories
About Elastic
Elastic, the Search AI Company, enables everyone to find the answers they need in real time, using all their data, at scale. Elastic’s solutions for search, observability, and security are built on the Elastic Search AI Platform — the development platform used by thousands of companies, including more than 50% of the Fortune 500.