11 hours ago
Remote, AustraliaStaff+
Responsibilities
- Own the architecture for measuring and improving AI agent quality across production traffic.
- Design cost- and accuracy-aware evaluation systems using deterministic checks, heuristics, and model-based judgment.
- Define benchmarks, datasets, evaluation environments, calibrated evaluators, and observability for agent experiments.
- Establish regression gates, pre-release checks, and production drift detection for agent, prompt, and model changes.
- Solve complex problems involving agent selection and routing, tool-use reliability, and autonomous agents without user feedback.
- Set technical direction and multi-quarter roadmaps across product, data science, central AI, and model-provider teams.
- Mentor engineers through design reviews and hands-on technical work.
Requirements
- 6+ years of experience in ML/AI engineering with deep expertise in LLMs, NLP, information retrieval, or recommendation.
- Proven experience building and shipping LLM-powered or agentic systems used by real customers and improving them through measurement.
- Strong evaluation and experimentation skills, including benchmark design, label and judge-quality assessment, leakage detection, and analysis of confounding factors.
- Strong software and data engineering skills, including coding, data pipelines, model serving, and production observability.
- Strong product judgment and the ability to select among deterministic, classical ML, and LLM-based approaches.
- Experience influencing technical direction across multiple teams and mentoring senior engineers.
- Regular use of AI tools and the ability to translate new research into production results.
- Sound judgment across quality, latency, cost, reliability, privacy, and access-control trade-offs.
- Experience with customer-configured agent platforms, trajectory-level evaluation of multi-step agents, prompt or agent optimization, enterprise SaaS, or a master's/PhD in a relevant field is preferred but not required.
Benefits
- Remote options are available; the team is based in Sydney and Melbourne.
- Opportunity to work on applied ML and production agentic systems under Atlassian's Rovo brand.
- Collaboration with product, data science, central AI teams, and model providers.
Categories
About Atlassian
Atlassian powers the collaboration that helps teams accomplish what would otherwise be impossible alone. From space missions and motor racing to bugs in code and IT requests, no task is too large or too small with the right team, the right tools, and the right practices. Over 300,000 global companies and 80% of the Fortune 500 rely on Atlassian’s software, like Jira, Confluence, Loom, and Trello, to help their teams work better together and deliver quality results on time. With our 300,000+ customers and team of 10,000+ Atlassians, we are building the next generation of team collaboration and productivity software. We believe the power of teams has the potential to change the world — one that is more open, authentic, and inclusive.