2 months ago
Base Salary
$200k - $350k/yr
Responsibilities
- Build the evaluation and optimization loop for Console’s core agents using production usage to improve quality, latency, and reliability.
- Improve agent behavior across prompts, programs, routing logic, constraints, and model adaptations using techniques such as DSPy and GEPA.
- Fine-tune, adapt, and evaluate specialist models for repeatable, high-volume agent tasks with production feedback or verifiable quality signals.
- Work across traces, evaluation infrastructure, agent orchestration, product workflows, and customer-facing AI behavior.
- Collaborate with product, engineering, and leadership to ship research-driven improvements into production and help define the company’s research function.
Requirements
- Strong technical background in software engineering, machine learning, or applied AI demonstrated through an advanced degree or equivalent experience building production AI systems.
- Strong software engineering fundamentals and sound judgment for designing, building, and debugging complex systems.
- Experience building AI evaluations involving datasets, judges, metrics, offline replay, tracing, or regression testing.
- Practical understanding of model adaptation and post-training methods, including LoRA/QLoRA, SFT, distillation, preference optimization, reward modeling, DPO/GRPO, and reinforcement learning from verifiable feedback.
- Ability to own projects end-to-end and ship measurable improvements quickly from real production usage.
- Interest in following current research on models, agent architectures, evaluations, post-training methods, and optimization techniques.
Benefits
- Comprehensive health, dental, and vision insurance
- Unlimited PTO
- 401(k)
- Equity
- Meals provided daily in the office
- Full benefits and competitive pay
- In-office work arrangement
Categories
About Console
Console is an AI-Native ITSM, built to help teams automate support and shift from doing the work to managing the system that does the work for them. From password resets to access requests, Console handles 60–80% of support requests directly in Slack or Teams before they ever reach a human.