Staff Software Engineer - AIOps
Early Warning Services2 months ago
Chicago, IL, USA +2 moreStaff+
Base Salary
$124k - $198k/yr
Responsibilities
- Design, build, test, and operate the production AIOps agent runtime, including event ingestion, context assembly, planning, tool invocation, verification, escalation, and response handling.
- Develop scoped, typed, versioned, auditable tool interfaces for CI/CD, infrastructure automation, Kubernetes, network services, ITSM, observability, and secrets-management systems.
- Implement safety controls including approvals, permission boundaries, blast-radius limits, dry runs, reversibility, read-only capabilities, and emergency shutdowns.
- Build event routing, telemetry, model routing, session and state management, retries, timeouts, cost and latency controls, and comprehensive auditability.
- Develop agent evaluation capabilities using incident replay, shadow-mode testing, proposed-action measurement, regression testing, and feedback from human approvals and operational outcomes.
- Create reusable agent capabilities, integrations, APIs, documentation, and self-service implementation patterns for other technology teams.
- Develop the operator experience, including approval workflows, activity and decision history, telemetry views, and APIs exposing agent state.
- Partner with engineering, operations, architecture, security, risk, and compliance stakeholders to establish reliable, secure, auditable, and regulatory-compliant solutions.
- Define and monitor platform effectiveness measures such as recommendation quality, approval and override rates, action success, rollback frequency, latency, cost, adoption, and operational outcomes.
Requirements
- Bachelor's degree in computer science, engineering, or a related technical field, or typically eight or more years of related experience in software engineering, platform engineering, distributed systems, DevOps, AIOps, or a related discipline.
- Eight or more years of related experience and demonstrated strength in software engineering and distributed-systems design.
- Strong production development experience with Python for services, APIs, automation, orchestration systems, or agent runtimes.
- Hands-on experience developing production agentic systems, tool-using AI applications, or autonomous or semi-autonomous software control loops beyond proof-of-concept implementations.
- Experience designing AI-agent tool-use or function-calling interfaces using MCP or comparable integration patterns.
- Experience implementing production controls for AI or automated systems, including evaluation, guardrails, monitoring, cost and latency management, and handling unintended or failed actions.
- Experience with event-driven architectures, messaging, production observability, structured logging, metrics, and distributed tracing.
- Experience developing front-end solutions with React, Next.js, or Angular and backend or API solutions with TypeScript or Python.
- Ability to lead complex technical work, make implementation decisions within established standards, collaborate across engineering and operations, influence without direct authority, and document architecture and controls.
- Background and drug screen required.
- Preferred qualifications include LangGraph or comparable orchestration experience, Go development, CI/CD, infrastructure automation, Kubernetes operations, ITSM, technology operations, anomaly detection, time-series analysis, chaos engineering, simulation, historical incident replay, game-day testing, regulated-control frameworks, FinTech or payments experience, and relevant technical certifications.
Benefits
- Hybrid work model for positions in Scottsdale, San Francisco, Chicago, or New York.
- Base pay varies by location, with stated annual ranges of $124,000-$165,000 in Phoenix/Chicago and $149,000-$198,000 in San Francisco, plus a discretionary incentive plan.
- Medical, dental, and vision coverage, with HSA contributions and flexible spending accounts.
- 401(k) plan with a 100% company Safe Harbor Match on the first 6% deferred, immediately upon eligibility.
- Flexible time off for exempt employees or generous PTO for non-exempt employees, 11 paid company holidays, and a paid volunteer day.
- Twelve weeks of paid parental leave and Maven Family Planning support for fertility, adoption, surrogacy, pregnancy, postpartum, and related services.
Tech Stack
Categories
About Early Warning Services
Early Warning Services builds payment networks and fraud/risk mitigation solutions for U.S. banks and credit unions. It operates the Zelle peer-to-peer payments network and is rolling out Paze, a digital wallet, alongside identity verification and account risk services. Headquartered in Scottsdale, Arizona, it is owned by a consortium of major U.S. banks and partners with thousands of financial institutions nationwide.