Staff Software Engineer - AIOps
Early Warning Services2 hours ago
Tokyo, JapanStaff+
Base Salary
$124k - $198k/yr
Responsibilities
- Design, build, test, and operate the production AIOps agent runtime, including event ingestion, context assembly, planning, tool invocation, verification, escalation, and response handling.
- Develop scoped, typed, versioned, auditable tool interfaces for CI/CD, infrastructure automation, Kubernetes, network services, ITSM, observability, and secrets-management systems.
- Implement safety controls including approvals, permission boundaries, blast-radius limits, dry runs, reversibility, read-only capabilities, and emergency shutdowns.
- Build event routing, telemetry, model routing, session and state management, retries, timeouts, cost and latency controls, and comprehensive auditability.
- Develop agent evaluation capabilities using incident replay, shadow-mode testing, proposed-action measurement, regression testing, and feedback from human approvals and operational outcomes.
- Create reusable agent capabilities, integrations, APIs, documentation, and self-service implementation patterns for other technology teams.
- Develop the operator experience, including approval workflows, activity and decision history, telemetry views, and APIs exposing agent state.
- Partner with engineering, operations, architecture, security, risk, and compliance stakeholders to establish reliable, secure, auditable, and regulatory-compliant solutions.
- Define and monitor platform effectiveness measures such as recommendation quality, approval and override rates, action success, rollback frequency, latency, cost, adoption, and operational outcomes.
Requirements
- Bachelor's degree in computer science, engineering, or a related technical field, or typically eight or more years of related experience in software engineering, platform engineering, distributed systems, DevOps, AIOps, or a related discipline.
- Eight or more years of related experience and demonstrated strength in software engineering and distributed-systems design.
- Strong production development experience with Python for services, APIs, automation, orchestration systems, or agent runtimes.
- Hands-on experience developing production agentic systems, tool-using AI applications, or autonomous or semi-autonomous software control loops beyond proof-of-concept implementations.
- Experience designing AI-agent tool-use or function-calling interfaces using MCP or comparable integration patterns.
- Experience implementing production controls for AI or automated systems, including evaluation, guardrails, monitoring, cost and latency management, and handling unintended or failed actions.
- Experience with event-driven architectures, messaging, production observability, structured logging, metrics, and distributed tracing.
- Experience developing front-end solutions with React, Next.js, or Angular and backend or API solutions with TypeScript or Python.
- Ability to lead complex technical work, make implementation decisions within established standards, collaborate across engineering and operations, influence without direct authority, and document architecture and controls.
- Background and drug screen required.
- Preferred qualifications include LangGraph or comparable orchestration experience, Go development, CI/CD, infrastructure automation, Kubernetes operations, ITSM, technology operations, anomaly detection, time-series analysis, chaos engineering, simulation, historical incident replay, game-day testing, regulated-control frameworks, FinTech or payments experience, and relevant technical certifications.
Benefits
- Hybrid work model for positions in Scottsdale, San Francisco, Chicago, or New York.
- Base pay varies by location, with stated annual ranges of $124,000-$165,000 in Phoenix/Chicago and $149,000-$198,000 in San Francisco, plus a discretionary incentive plan.
- Medical, dental, and vision coverage, with HSA contributions and flexible spending accounts.
- 401(k) plan with a 100% company Safe Harbor Match on the first 6% deferred, immediately upon eligibility.
- Flexible time off for exempt employees or generous PTO for non-exempt employees, 11 paid company holidays, and a paid volunteer day.
- Twelve weeks of paid parental leave and Maven Family Planning support for fertility, adoption, surrogacy, pregnancy, postpartum, and related services.
Tech Stack
Categories