4 hours ago
Remote, CanadaStaff+
Responsibilities
- Set technical direction for AI/ML platform and AI evaluation platform systems spanning multiple teams.
- Build and operate model training, model serving, feature and data workflows, orchestration, deployment, observability, and evaluation tooling.
- Develop scalable backend services and execution infrastructure for GenAI-powered applications.
- Build evaluation capabilities including trace ingestion, scorer infrastructure, LLM-as-judge calibration, regression gates, and self-service tooling.
- Partner with applied scientists, product engineering teams, senior engineers, and leadership on roadmaps and build/buy/adopt decisions.
- Drive projects to completion with measurable business impact and lead incident response for production systems.
Requirements
- 10+ years of professional software engineering experience with staff-level technical leadership or equivalent scope.
- 3+ years of production experience building AI/ML infrastructure such as model serving, feature platforms, training systems, orchestration systems, or ML observability.
- Hands-on production experience with LLM-powered systems, including areas such as LLM gateways, prompt and model lifecycle management, RAG, agentic architectures, or LLM cost optimization.
- Experience building or operating AI evaluation systems, including offline evaluation pipelines, LLM-as-judge calibration, regression testing, or A/B measurement of AI features.
- Deep experience designing, building, and operating reliable distributed systems at meaningful scale, including on-call and incident leadership.
- Ability to use AI coding tools and validate, critique, and refine AI-generated output.
Categories
About Thumbtack
Thumbtack builds a marketplace and app that connects U.S. homeowners with service professionals for repairs, maintenance, and home improvements. Pros pay for leads and bookings generated on the platform, and can use built-in tools for scheduling, messaging, and payments. Founded in 2008 and headquartered in San Francisco, it features a network of about 300,000 local businesses and has supported over 11 million projects nationwide.