Software Engineer, Inference
Thinking Machines Lab5 hours ago
Base Salary
$300k - $400k/yr
Responsibilities
- Operate and scale production inference systems serving live traffic, including Tinker's multi-tenant serving platform.
- Own safe, incremental rollouts for new models, model versions, and inference optimizations.
- Build and improve observability, monitoring, alerting, and capacity-planning systems.
- Partner with inference and research teams to productionize new serving techniques.
- Lead incident response, root-cause analysis, and durable fixes for production inference issues.
- Design graceful degradation, failover, and redundancy for resilient serving.
- Manage capacity and cost tradeoffs as traffic and model sizes scale.
Requirements
- Experience operating large-scale, latency-sensitive production systems.
- Proficiency in Python and Go or another systems language.
- Experience with observability, monitoring, and incident response for production services.
- Strong understanding of distributed systems and failure at scale.
- Preferred: experience running production inference for large language models or other large-scale ML systems.
- Preferred: experience with canarying, blue/green deployments, feature flags, capacity planning, GPU or TPU cost optimization, batching, caching, or quantization.
- Comfort with on-call responsibilities, leading critical incident response, and working autonomously in a fast-changing environment.
Benefits
- Based in San Francisco, California.
- Annual salary range of $300,000–$400,000 USD.
- Visa sponsorship is available, with the company committed to working through the visa process for the right fit, though success is not guaranteed for every candidate or role.
- Generous health, dental, and vision benefits, unlimited PTO, paid parental leave, and relocation support as needed.
Categories
About Thinking Machines Lab
Thinking Machines Lab develops AI and generative AI software and conducts applied research to help organizations make data-driven decisions. The company builds products and data science solutions for enterprise use cases, pairing foundational models with practical tooling and services across industries. It is privately held and headquartered in San Francisco.