Senior Software Engineer - Observability & IRM
The Trade Desk4 hours ago
Base Salary
$125k - $250k/yr
Responsibilities
- Build and maintain automation for alerting, escalation, incident channels, retrospectives, and SLA tracking.
- Develop and maintain incident management tooling.
- Help evaluate and migrate the logging stack, including re-evaluating the logging vendor and collection architecture.
- Extend the Backstage service catalog with Kubernetes integrations, maturity models, and SLO adoption tooling.
- Build alert-quality systems with smarter routing, better grouping, and improved feedback loops between alerts and owning teams.
- Support engineers across the company in detecting, managing, and optimizing production incidents.
Requirements
- Experience building and operating production infrastructure or internal developer tooling.
- Comfort working across distributed systems, Kubernetes, observability pipelines, and web-based tooling.
- Familiarity with logging, alerting, and on-call workflows.
- Strong debugging instincts and the ability to respond when production systems break.
- Clear communication skills for explaining tradeoffs and advocating for solutions.
- Experience with Grafana, Prometheus, or similar observability tools is a plus.
- Familiarity with Sumo Logic or other log management platforms is a plus.
- Prior work with developer portals or service catalog tooling such as Backstage or OpsLevel is a plus.
- Experience with Kubernetes at scale is a plus.
- Deep understanding of HunnyPt is a plus.
Benefits
- Comprehensive healthcare, including medical, dental, and vision, with premiums fully paid for employees and dependents.
- 401(k) retirement plan with company match, short- and long-term disability coverage, and basic life insurance.
- Well-being benefits, tuition reimbursement, parental leave, sick time, vacation, and approximately 13 paid holidays per year.
- Employee Stock Purchase Plan allowing employees to purchase company stock at a discount.
Tech Stack
Categories
DevOpsSite Reliability