Senior Site Reliability Engineer - Linux Systems & Application Observability
tastytrade7 hours ago
Base Salary
$180k - $200k/yr
Responsibilities
- Build self-healing, fault-tolerant infrastructure and internal tooling that automates operational work and reduces toil.
- Analyze and improve telemetry, logging, alerting, and observability coverage so failures are detected before affecting customers.
- Own capacity planning, load testing, scalability improvements, and architectural bottleneck identification across the HashiCorp Nomad service fabric.
- Extend observability with Prometheus, Honeycomb, OpenTelemetry, and direct service instrumentation.
- Define SLOs, error budgets, and multi-window burn-rate alerting for critical brokerage flows.
- Participate in production on-call operations and build a blameless post-incident review process.
- Mentor engineers across teams and establish a culture of site reliability champions.
Requirements
- Hands-on experience designing and shipping fault-tolerant, self-healing distributed systems.
- Deep understanding of distributed systems, Linux systems, cloud-native architectures, or containerization.
- Experience identifying gaps in observability and telemetry systems, including instrumentation, alerting, and visibility.
- Production experience scaling systems under load through capacity planning and architectural bottleneck analysis.
- Hands-on experience with OpenTelemetry, Prometheus, and Grafana, including direct service instrumentation.
- Strong Linux internals and networking fundamentals, including TCP/IP, UDP/multicast, packet capture, and flow analysis.
- Production on-call experience and comfort establishing blameless post-incident review practices.
- Working knowledge of SLOs and error budgets; HashiCorp Nomad, Consul, or Vault experience is a strong plus.
- Strong programming skills in Python, Ruby, Java, or a similar language.
Benefits
- Chicago hybrid work arrangement with 3 days per week in the office.
- Performance bonuses and stock purchase options.
- Medical, vision, dental, 401(k), paid vacation, and paid sick days.
- Gym membership reimbursement, in-building gym, commuter benefits, and shuttle service to and from Metra.
- Pet insurance, wellness and mental health programs, charitable donation matching, and two paid volunteer days off.
- Daily catered lunch and a kitchen with snacks and beverages when in the office.