Software Engineer, Inference
Thinking Machines Lab1 month ago
Base Salary
$300k - $400k/yr
Responsibilities
- Operate and scale production inference systems serving live traffic, including Tinker’s multi-tenant serving platform.
- Own safe, incremental rollouts for new models, model versions, and inference optimizations.
- Build observability, alerting, and capacity-planning systems for rapid detection and resolution of production issues.
- Partner with inference and research teams to productionize new serving techniques while maintaining reliability.
- Lead incident response, root-cause analysis, and durable fixes for production inference issues.
- Design graceful degradation, failover, and redundancy strategies for resilient model serving.
- Manage capacity and cost tradeoffs as traffic and model sizes grow.
Requirements
- Experience operating large-scale, latency-sensitive production systems.
- Proficiency in Python and Go or another systems language.
- Experience with observability, monitoring, and incident response for production services.
- Strong understanding of distributed systems and failure modes at scale.
- Preferred: experience running production inference for large language models or other large-scale ML systems.
- Preferred: experience with deployment and rollout systems such as canarying, blue/green deployments, or feature flags.
- Preferred: experience with capacity planning and cost optimization for GPU or TPU infrastructure.
- Preferred: familiarity with inference techniques such as batching, caching, or quantization and their operational implications.
- Ability to participate in on-call rotations and lead incident response for critical production systems.
- Ability to work with high autonomy in a fast-changing, early-stage environment.
Benefits
- Based in San Francisco, California.
- The company sponsors visas and is committed to working through the visa process for qualified candidates, although success is not guaranteed for every candidate or role.
- Generous health, dental, and vision benefits.
- Unlimited paid time off.
- Paid parental leave.
- Relocation support as needed.
Categories
ML EngineeringSite Reliability
About Thinking Machines Lab
Thinking Machines Lab develops AI and generative AI software and conducts applied research to help organizations make data-driven decisions. The company builds products and data science solutions for enterprise use cases, pairing foundational models with practical tooling and services across industries. It is privately held and headquartered in San Francisco.