5 days ago
Remote, United StatesSenior
Base Salary
$200k - $250k/yr
Responsibilities
- Design, build, and own AI evaluation systems using golden datasets, LLM-as-judge methods, automated scorers, regression detection, and quality tracking.
- Build and evolve the core agent harness and AI systems that optimize performance against evaluations.
- Develop automated self-improvement loops in which agents modify other agents or prompts, rerun evaluations, and iterate.
- Benchmark models and manage model selection, routing, and fallback strategies across multiple providers while balancing quality, latency, and cost.
- Architect and implement scalable AI platform services for model deployment, orchestration, and lifecycle management.
- Build backend APIs and systems that integrate AI capabilities into ClickUp’s core platform.
- Develop model-serving, monitoring, logging, and automated-evaluation infrastructure for reliable production AI services.
- Drive AI privacy, security, compliance, anonymization, and secure data-handling practices.
- Collaborate with product, frontend, data, and engineering teams to deliver reliable AI experiences.
Requirements
- At least 5 years of software engineering experience, including at least 2 years building production LLM or ML systems.
- Production experience building LLM evaluation systems with golden datasets, LLM-as-judge, judge calibration, offline and online evaluations, and product-quality measurement.
- Strong experiment design and statistical rigor, including A/B testing, offline-versus-online metrics, and regression analysis.
- Experience applying LLMs and AI models to customer-facing product features.
- Experience integrating and managing multiple LLMs, including benchmarking, routing, and cost, latency, and quality tradeoffs.
- Deep backend engineering, distributed systems, and cloud-native technology expertise.
- Strong programming skills in Python, Go, TypeScript, or similar backend and AI-platform languages.
- Proficiency with orchestration frameworks and workflow engines such as LangGraph, Airflow, or Ray.
- Familiarity with search and retrieval technologies in AI-driven applications.
- Strong collaboration and communication skills.
- Preferred experience in data science, applied science, or machine learning engineering.
- Preferred experience with Braintrust, LangSmith, Weave, Arize, or in-house evaluation frameworks, as well as model routing, AI gateways, or inference optimization.
Benefits
- Remote work arrangement, indicated by the #LI-REMOTE listing.
- Visa sponsorship for this engineering role is not guaranteed and depends on the specific role’s business needs.
Tech Stack
Categories
About ClickUp
ClickUp builds an all-in-one work management platform for teams, combining tasks, docs, chat, whiteboards, calendars, time tracking, and AI features in a single workspace. It sells SaaS subscriptions with a freemium tier to individuals, SMBs, and enterprises, and integrates with common productivity tools. Founded in 2017 and headquartered in San Diego, the privately held company reports millions of users across project management, collaboration, and knowledge workflows.