12 hours ago
Remote, AustraliaSenior
Responsibilities
- Build offline evaluation suites and benchmarks using real usage patterns, synthetic scenarios, and simulated tools.
- Build online quality metrics from production traffic, including behavioral signals and model-based judging of agent trajectories.
- Calibrate judges against human ratings and independent outcome signals, and evolve evaluations as agents improve.
- Use evaluation results and production signals to improve prompts, context selection, tool use, retrieval, routing, and overall agent quality.
- Improve quality while reducing cost and latency across production LLM systems.
- Own projects from problem definition through experimentation, deployment, and ongoing measurement.
- Contribute to design and code reviews, coach engineers, and help software engineers write effective evaluations.
Requirements
- Bachelor's or master's degree in Computer Science or a related field, or equivalent practical experience.
- 4+ years of related industry experience in machine learning, including hands-on experience designing, evaluating, and operating LLM-based or agentic systems.
- Strong coding skills in Python and/or Kotlin and the ability to write performant, production-quality code using AI tools.
- Experience designing experiments, building evaluation datasets and benchmarks, and analyzing model and system behavior.
- Familiarity with SQL, distributed data processing, and cloud data environments.
- Experience building and operating production ML systems with attention to quality, reliability, and cost.
- Regular use of AI tools in engineering workflows, with judgment to validate outputs and take ownership of results.
- Ability to lead ambiguous technical work from problem definition through delivery and communicate ML trade-offs clearly.
- Preferred experience with agent evaluations, trajectory analysis, human evaluation, LLM-as-a-judge, tool-using agents, retrieval-augmented generation, retrieval, ranking, recommendation, enterprise SaaS, or large-scale product environments.
Categories
About Atlassian
Atlassian powers the collaboration that helps teams accomplish what would otherwise be impossible alone. From space missions and motor racing to bugs in code and IT requests, no task is too large or too small with the right team, the right tools, and the right practices. Over 300,000 global companies and 80% of the Fortune 500 rely on Atlassian’s software, like Jira, Confluence, Loom, and Trello, to help their teams work better together and deliver quality results on time. With our 300,000+ customers and team of 10,000+ Atlassians, we are building the next generation of team collaboration and productivity software. We believe the power of teams has the potential to change the world — one that is more open, authentic, and inclusive.