Senior Software Engineer II - LLM Observability
Honeycomb.io16 hours ago
Remote, CanadaSenior
Responsibilities
- Own end-to-end development of AI observability capabilities from ingestion to in-product visualizations.
- Architect and evolve the data model and infrastructure for observing agents and their ecosystems.
- Build systems that surface AI-specific signals including cost, token usage, errors, and anomalies.
- Collaborate with product managers and designers to create usable product workflows.
- Teach best practices for exploring and understanding AI system behavior across conversations and workloads.
- Iterate through experimentation and fast feedback loops while sharing successes and failures.
- Participate in the on-call rotation after approximately 90 days.
Requirements
- Significant software engineering experience with a record of shipping and owning features end to end.
- Experience shipping customer-facing features at a SaaS company rather than only internal tooling.
- Full-stack proficiency with backend development in Go and/or Python and frontend development in TypeScript and React.
- Experience working closely with product managers and designers.
- Regular hands-on use of AI tools in day-to-day engineering work.
- Experience being on call.
- Ability to navigate ambiguity and shifting requirements in a fast-moving field.
- Preferred experience building LLM- or agent-powered features in production.
- Preferred hands-on experience with observability tooling such as OpenTelemetry.
Benefits
- Fully distributed, remote work arrangement.
- Generous equity and an employee-friendly stock program.
- Unlimited paid time off.
- Home office, co-working, and internet stipend.
- Full benefits coverage for employees, with additional dependent coverage available.
- Up to 16 weeks of paid parental leave regardless of path to parenthood.
- Annual development allowance.
Tech Stack
Categories
About Honeycomb.io
Honeycomb.io builds a SaaS observability platform for software engineering, SRE, and DevOps teams to debug and operate distributed systems using tracing, event data, and high-cardinality queries. Its product integrates with OpenTelemetry and supports SLOs, instrumentation, and production debugging workflows. Founded in 2016 and headquartered in San Francisco, the company is privately held and counts customers such as Slack, HelloFresh, LaunchDarkly, and Vanguard.