AI Agent Architect
Emergent Labs3 days ago
Bengaluru, IndiaSenior
Responsibilities
- Analyze production agent behavior to identify failure modes, regressions, bottlenecks, and quantitative signals.
- Run experiments involving prompt tuning, evaluation datasets, experimentation, and harness engineering to improve agent quality and code outcomes.
- Build evaluation frameworks, metrics, datasets, validation processes, and dashboards for measuring agent quality at scale.
- Manage evaluation gates, staged rollouts, and explicit rollback criteria for production changes.
- Drive context engineering, memory systems, tool use, and long-horizon execution improvements.
- Make evidence-based decisions about regressions, benchmark overfitting, noisy results, and whether to ship or stop an approach.
Requirements
- 5+ years of experience building and shipping software with end-to-end ownership.
- At least 1 year of hands-on experience with agentic systems and their evaluation.
- Hands-on experience with experimentation, metrics, debugging, and analysis using Python, SQL, or similar tools.
- Strong systems thinking, product judgment, and ability to reason about noise, confounds, distribution shift, and selection effects.
- Deep current interest in LLMs, agents, and emerging model capabilities.
- Ability to independently scope work, demonstrate leadership presence, and challenge weak ideas while maintaining rigorous measurement.
- The role is ideally suited to principal or staff individual contributors, software architects, lead builders, and technical cofounders.
Benefits
- Lunch and dinner are provided daily.
- Family insurance provides 5 Lakhs of coverage for the employee and family.
- Unlimited paid time off is provided.
- Flexible working hours and arrangements are available.
Categories