Senior Software Engineer, AI Job Orchestration
Firmus Technologies2 months ago
Sydney, AustraliaSenior
Responsibilities
- Design and build unified job submission APIs, CLI, and web interfaces for AI training, inference, and fine-tuning workloads across Kubernetes and Slurm.
- Implement job metadata schemas, versioning, authentication and authorization, RBAC, resource quotas, and multi-tenant isolation.
- Build scheduling and orchestration capabilities covering priority classes, preemption, fairness, quota enforcement, and intelligent job routing.
- Create a catalog of training, inference-serving, and fine-tuning templates with discovery, validation, and manifest generation.
- Integrate job submissions with observability pipelines and expose telemetry APIs for GPU metrics, cost accrual, training MFU, inference latency, throughput, and tokenomics.
- Extend orchestration for inference specifications and integrate with inference-serving APIs.
- Collaborate with platform, infrastructure, LLM engineering, and ModelOps teams on dashboards, templates, reliability standards, and API contracts.
Requirements
- 5–7 years of backend engineering experience building production APIs and distributed systems using Python, Go, or Java.
- Deep Kubernetes expertise, including Job controllers, Pod specifications, resource requests and limits, RBAC, network policies, and debugging.
- Hands-on Slurm experience with job submission, resource allocation, job queues, and sbatch scripting.
- Strong knowledge of distributed systems, scheduling algorithms, fairness, preemption, and resource management.
- Experience designing data models and schemas, handling versioning and migrations, and maintaining backward compatibility.
- Experience with observability, logging, tracing, production troubleshooting, streaming APIs, real-time webhooks, and system-level integration patterns.
- Experience shipping job scheduling or workflow systems at scale and designing multi-tenant resource allocation and data isolation.
- Ability to design intuitive and extensible RESTful or gRPC APIs and collaborate across infrastructure, platform, LLM engineering, and ModelOps teams.
Benefits
- Full-time employment.
- Role based in either Sydney, Australia, or Singapore.
- Commitment to a diverse and inclusive workplace.
Tech Stack
Categories