
Senior Software Engineer, Reliability
Temporal Technologies1 day ago
Base Salary
$141k - $231k/yr
Responsibilities
- Own reliability outcomes for operating Temporal Cloud end to end across engineering, infrastructure, and product.
- Build and evolve a Reliability Agent that gathers context from telemetry, incidents, runbooks, deployments, and service history.
- Design safe agent experiences with human oversight, escalation paths, auditability, and safeguards against incorrect actions.
- Lead incident detection, triage, escalation, mitigation, communication, and post-incident reviews through corrective-action completion.
- Define reliability practices including service-level objectives, alerting thresholds, operational readiness criteria, and escalation paths.
- Plan and run gamedays, expand chaos testing, lead load and capacity testing, and improve observability standards.
- Build reliability scorecards covering service health, operational readiness, agent effectiveness, and improvement priorities.
- Mentor others, share operational knowledge, align stakeholders, and participate in an on-call rotation.
Requirements
- Strong computer science fundamentals in distributed systems, concurrency, performance, and fault-tolerant design.
- Experience designing, building, or operating complex systems under high load and partial failure.
- Experience building AI-powered tools, agents, or production automation, including tool use, evaluation, state, failure modes, and human-in-the-loop design.
- Demonstrated success driving reliability improvements across multiple services, systems, or engineering teams.
- Hands-on incident-response and post-incident-review experience, including leading incidents and completing corrective actions.
- Hands-on experience with gamedays, chaos testing, load testing, capacity planning, reliability scorecards, or observability programs.
- Sound judgment in ambiguous situations and the ability to balance reliability, agent autonomy, safety, customer impact, and delivery speed.
- Clear communication skills, stakeholder-alignment ability, mentoring experience, and willingness to participate in on-call.
- Nice to have experience with multi-tenant systems, noisy-neighbor protections, internal platforms, LLM observability, agent evaluation, retrieval-augmented generation, workflow orchestration, or relevant open-source contributions.
Benefits
- Remote role based in the United States with no regular travel expected.
Categories
Site Reliability
About Temporal Technologies
Temporal Technologies builds an open-source durable workflow platform that software teams use to orchestrate long-running, stateful application logic across services. It monetizes through a managed cloud offering and commercial support for its server and SDKs. Founded in 2019 and headquartered in Bellevue, Washington, it lists public adopters such as Netflix, Snap, Indeed, and Datadog.