6 hours ago
Sydney, Australia +2 moreSenior
Responsibilities
- Design end-to-end LLM evaluation plans, metric systems, and rubrics for dialogue, financial trading, and other business scenarios.
- Lead evaluation dataset, benchmark, annotation guideline, and quality-control design.
- Analyze model capability boundaries and failure modes and translate findings into actionable improvement recommendations.
- Automate and scale evaluation workflows by building evaluation platforms and toolchains.
- Collaborate with algorithm, product, and data teams to establish evaluation standards and drive model iteration.
Requirements
- Master's degree or above in Computer Science, Artificial Intelligence, Mathematics, Statistics, or a related field.
- Hands-on LLM evaluation experience at a large technology company, including commercial deployment evaluation.
- Understanding of LLM principles, training, and fine-tuning processes.
- Experience with human evaluation, model-based automatic evaluation, LLM-as-a-judge methods, and metric computation.
- Ability to define evaluation dimensions, write discriminative rubrics, and ensure data representativeness, annotation consistency, and result reliability.
- Proficiency in Python and experience with evaluation workflow automation, benchmark construction, evaluation platform development, data processing, scripting, and result analysis.
- Strong business understanding and communication skills for cross-team collaboration.
- Preferred experience with dialogue systems, AI agents, financial or trading LLMs, AI training or evaluation data, annotation systems, RLHF, reward models, or preference data.
Benefits
- Competitive salary and company benefits.
- Work-from-home arrangement, which may vary based on the business team's work.
- Opportunities for career growth and continuous learning.
- Collaboration with global talent in a flat, user-centric organization.
- Autonomy on innovative, fast-paced projects.
Tech Stack
Categories
About Binance
Binance builds a global cryptocurrency exchange and related infrastructure for retail and institutional traders, developers, and businesses. Its products include spot and derivatives trading, fiat on-ramps, wallets, payments, staking, custody, and Web3/NFT services, with revenue driven largely by trading and transaction fees. Founded in 2017 and privately held, it operates worldwide and also stewards the BNB Chain ecosystem, with its main exchange regularly ranking first by crypto trading volume.