Microsoft

Principal Software Engineer - AI Inference and Serving (CoreAI)

Microsoft
Apply
1 day ago
Redmond, WA, USAStaff+
H1B sponsor

Base Salary

$143k - $331k/yr

Responsibilities

  • Own an inference service and lead delivery across gateway and runtime boundaries.
  • Evaluate prefix caching, prefill/decode disaggregation, speculative decoding, quantization, and tensor and expert parallelism.
  • Preserve streaming, cancellation, retries, token-accounting correctness, model quality and numerical correctness.
  • Lead capacity planning, incident prevention, model onboarding and operational support while mentoring engineers.
  • Build secure, tenant-isolated systems with automated testing, observability, reliable recovery and safe releases.
  • Use reproducible benchmarks and production feedback to improve throughput, latency, capacity and cost.
  • Build provider-aware serving stacks and integrate with training and post-training stacks for OpenAI, Anthropic, Microsoft AI and open-source models.

Requirements

  • Bachelor's degree in computer science or a related technical field and at least 6 years of technical engineering experience with coding in C, C++, C#, Java, JavaScript or Python, or equivalent experience.
  • Preferred: 8 or more years of technical engineering experience with coding in C, C++, C#, Java, JavaScript or Python, or equivalent experience.
  • Preferred: proven distributed-services and model-serving delivery spanning at least two connected layers.
  • Preferred: strong software engineering fundamentals, debugging skills and cross-team communication.
  • Preferred: experience translating customer requirements into service objectives and measurable performance improvements.
  • Preferred: experience with containerized services, Kubernetes and serving runtimes such as vLLM, SGLang or TensorRT-LLM.
  • Ability to meet Microsoft, customer and/or government security screening requirements, including the Microsoft Cloud background check.
Microsoft

About Microsoft

10,000+ employees

Microsoft develops operating systems, productivity software, cloud services, developer tools, and consumer devices for individuals, enterprises, and governments. Its main products include Windows, Microsoft 365, Azure, Visual Studio/GitHub, Xbox, and LinkedIn; revenue comes from software subscriptions and licenses, cloud consumption, hardware sales, and advertising. Founded in 1975 and headquartered in Redmond, Washington, Microsoft is a public company traded on Nasdaq.

Contact me