3 months ago
Bengaluru, IndiaSenior
Responsibilities
- Design and own synthetic data generation pipelines for improving model performance on edge cases.
- Develop evaluation datasets for domain-specific languages and maintain automated data quality checks and research data pipelines.
- Define rubrics for evaluating AI outputs and build continuous monitoring, logging, and performance optimization systems.
- Create tooling for domain experts to refine and evolve training datasets.
- Lead solutions for long-context handling and cascading errors in LLM pipelines.
- Design and evaluate production-grade AI agent systems end to end.
- Drive prompt engineering strategy and experimentation frameworks across the team.
Requirements
- 4+ years of engineering experience, including at least 2 years focused on production LLM systems or applied ML.
- Deep hands-on experience with LLMs, synthetic data pipelines, evaluation frameworks, and AI system optimization.
- Strong Python skills; Go or TypeScript experience is a plus.
- Understanding of prompt engineering, model fine-tuning, and evaluation methodology.
- Experience building and operating production AI systems rather than prototypes.
- Strong debugging ability for complex, non-deterministic system failures.
- Experience with model monitoring, bias evaluation, or dataset management tooling is preferred.
- Background in semantic parsing, domain-specific languages, or structured output models is preferred.
- Experience with agentic systems, multi-step reasoning pipelines, or tool-use frameworks is preferred.
Benefits
- Full family health insurance
- Annual learning and development budget
- Dedicated mentor and coaching support
- Free snacks and dinner at the office
Tech Stack
Categories
About Metaforms
Metaforms builds an AI‑agent research operations platform that speeds up survey design, programming, testing, data cleaning/coding, tabulation, and reporting for market research agencies and in‑house insights teams. It sells a SaaS workflow that plugs into tools such as Qualtrics, Confirmit, Decipher, and Dimensions. Founded in 2023 and headquartered in New York, the company is privately held.