Senior Software Engineer, ML API – AI Inference
Cerebras Systems2 days ago
Responsibilities
- Design, implement, test, and operate inference features for chat completions, text generation, streaming, model configuration, tool calling, structured outputs, multimodal inputs, and emerging capabilities.
- Lead multi-component technical delivery by writing designs, evaluating tradeoffs, defining milestones, and coordinating with engineers and partner teams.
- Evolve compatible inference APIs with clear request and response semantics, validation, versioning, deprecation, error handling, and testing.
- Integrate foundation models, tokenizers, chat templates, generation configuration, sampling methods, and model-specific features into the serving platform.
- Extend inference services and integrate frameworks and runtimes including vLLM, PyTorch, Hugging Face libraries, LLGuidance, and Cerebras runtime components.
- Implement routing, state transfer, retries, cancellation, error handling, and request lifecycle management for disaggregated GPU-prefill and Cerebras-decode systems.
- Improve time to first token, streaming latency, throughput, and resource efficiency through optimization of tokenization, serialization, batching, scheduling, and communication.
- Build conformance tests, model-validation suites, numerical comparisons, integration tests, and performance benchmarks.
- Improve metrics, tracing, logging, alerts, diagnostics, resilience, and incident response for production inference services.
- Partner with product, support, solutions, customers, and developers to improve configuration, documentation, examples, and debugging workflows.
- Review code and designs, mentor less-experienced engineers, share domain knowledge, and remove delivery roadblocks.
Requirements
- At least 5 years of software engineering experience, including independent ownership of production services, distributed systems, or substantial software components.
- Strong Python programming skills and proficiency in Go, C++, Rust, or another language used for production services.
- Experience designing and delivering multi-component features, documenting designs, evaluating tradeoffs, and coordinating implementation.
- Experience building and evolving production APIs with validation, error handling, compatibility, and testing practices, plus familiarity with REST, gRPC, or streaming protocols.
- Experience integrating software across service, framework, runtime, or infrastructure boundaries.
- Experience operating Linux services with containers, Kubernetes or comparable orchestration systems, and CI/CD.
- Ability to diagnose correctness, reliability, and performance issues across distributed components and deliver measurable improvements.
- Strong communication and collaboration skills, including technical leadership on a small project and support for other engineers through reviews or mentorship.
- Bachelor’s degree in Computer Science, Computer Engineering, Electrical Engineering, or a related discipline, or equivalent practical experience.
- Preferred experience with production ML or LLM inference APIs, including OpenAI-compatible interfaces.
- Preferred experience extending or contributing to vLLM, SGLang, PyTorch, Hugging Face Transformers, Triton, TensorRT-LLM, or another ML systems project.
- Preferred knowledge of tokenization, chat templates, generation configuration, logits processing, stopping criteria, tool calling, structured generation, and constrained decoding.
- Preferred experience with disaggregated prefill/decode architectures, KV-cache transfer, prefix caching, chunked prefill, inference request scheduling, multi-model or multi-tenant serving, and performance-regression testing.
- Preferred familiarity with GPU inference, AMD Instinct accelerators, ROCm, CUDA, collective communication, multi-node model serving, reduced-precision inference, and quantization formats such as BF16, FP8, FP4, INT8, or INT4.
- Experience with SDKs, developer tools, configuration systems, self-service ML workflows, customers, or open-source communities is preferred.
Benefits
- Work on a breakthrough AI platform and one of the world’s fastest AI supercomputers.
- Opportunities to publish and open-source cutting-edge AI research.
- Startup vitality with stated job stability and a simple, non-corporate culture.
- Cerebras states that it is committed to an equal and diverse environment and supports continuous learning, growth, and employee support.
Categories
About Cerebras Systems
Cerebras Systems designs and sells AI compute systems built around its wafer-scale WSE-3 processor, delivered as the CS-3 appliance and via the Cerebras Cloud. It targets enterprises, model labs, and government users needing fast training and inference, and offers on‑prem and cloud deployments. Privately held and headquartered in Sunnyvale, California, the company announced a multi-year partnership with OpenAI to deploy large-scale inference capacity.