18 hours ago
Base Salary
$207k - $300k/yr
Responsibilities
- Analyze and optimize AI inference workloads across application, model, and distributed fleet infrastructure layers.
- Design and implement inference optimization techniques to increase throughput and reduce latency.
- Investigate and resolve complex model inference performance bottlenecks across the stack.
- Model latency-to-cost impacts of variables such as batch size and utilization goals and translate findings into production signals.
- Develop investigative tools and metrics, including compute and FLOPs funnels, to track fleet-wide compute usage.
Requirements
- Bachelor's degree in Computer Science, Computer Engineering, Electrical Engineering, Applied Mathematics, or a related technical field, or equivalent practical experience.
- 8 years of experience in software development.
- Experience with Python and C++, including navigating, debugging, and modifying serving codebases.
- Experience with AI model execution constraints, throughput-latency tradeoffs, memory bandwidth limitations, and modern serving architectures.
- Preferred experience with real-world LLM inference serving environments or open-source inference frameworks such as vLLM, TensorRT-LLM, SGLang, or Dynamo.
- Preferred experience profiling workloads with ML profilers such as PyTorch profiler and internal trace analysis tools.
- Preferred experience with observability and reliability for large distributed systems.
- Preferred familiarity with GPU, TPU, and accelerator performance concepts including memory bandwidth, quantization, collective communication, and kernels.
Benefits
- Equity and benefits are provided; the posting also references Google benefits information.
Categories
About Google
Google builds consumer and enterprise software and services including Search, Android, YouTube, Chrome, Maps, Gmail, and Google Cloud. Its business model centers on digital advertising and paid cloud, software, and hardware offerings (e.g., Pixel and Nest) for consumers, developers, and organizations. Founded in 1998 and headquartered in Mountain View, California, Google operates globally as a subsidiary of Alphabet Inc.