Microsoft

Software Engineer II & Senior Software Engineer - AI Inference and Serving (CoreAI)

Microsoft
Apply
1 day ago
Redmond, WA, USAMid Level / Senior
H1B sponsor

Base Salary

$102k - $261k/yr

Responsibilities

  • Own an AI inference serving component from design through production.
  • Profile execution, memory, and dependency bottlenecks and improve throughput, latency, and cost.
  • Improve cache-aware routing, admission control, continuous batching, and KV-cache management.
  • Validate time-to-first-token, tail latency, and cost per token through load and failure testing.
  • Build secure, tenant-isolated systems with automated testing, observability, recovery, and safe releases.
  • Provide operational support and improve services using production feedback.
  • Preserve model quality and numerical correctness through reproducible benchmarks.
  • Pursue near-linear throughput scaling across supported GPU and node configurations.
  • Build provider-aware serving stacks and integrate with training and post-training stacks for supported models.

Requirements

  • Bachelor's degree in computer science or a related technical field and at least 2 years of technical engineering experience with coding, or equivalent experience.
  • Preferred: at least 5 years of technical engineering experience with coding, or equivalent experience.
  • Proven coding experience in C++, C#, Rust, Go, Java, or Python, with networking and concurrency fundamentals.
  • Experience with software engineering fundamentals, debugging, and cross-team communication.
  • Experience shipping and operating gateways, model-serving runtimes, or accelerator software.
  • Experience with containerized services, Kubernetes, and serving runtimes such as vLLM, SGLang, or TensorRT-LLM.
  • Ability to meet Microsoft, customer, and/or government security screening requirements, including the Microsoft Cloud background check.
Microsoft

About Microsoft

10,000+ employees

Microsoft develops operating systems, productivity software, cloud services, developer tools, and consumer devices for individuals, enterprises, and governments. Its main products include Windows, Microsoft 365, Azure, Visual Studio/GitHub, Xbox, and LinkedIn; revenue comes from software subscriptions and licenses, cloud consumption, hardware sales, and advertising. Founded in 1975 and headquartered in Redmond, Washington, Microsoft is a public company traded on Nasdaq.

Contact me