2 months ago
Canada (city not specified)Staff+
Responsibilities
- Own the production improvement loop for Context Engine by diagnosing agent, automation, retrieval, and memory failure modes and measuring fixes.
- Define safe versioning, rollout, regression coverage, staged evaluation, shadow evaluation, and guardrails for prompts, skills, and automations.
- Design telemetry for agent traces, tool calls, and knowledge retrieval, and use it to support evaluations, dashboards, and feedback loops.
- Partner with data scientists on golden datasets and evaluators for quality, latency, and cost.
- Run product experiments end to end, including instrumentation, unattended execution, and interpretation of results.
- Review designs and pull requests, mentor engineers in eval-driven development, and write technical proposals that shape the roadmap.
Requirements
- 10+ years of software engineering experience, including recent experience shipping and operating AI-driven products on real production traffic.
- Track record of eval-driven product improvement using agent or LLM traces, user feedback, evaluation design, production fixes, and outcome measurement.
- Direct experience building agents with state and memory and safely iterating on prompts, skills, and tool behavior in production.
- Familiarity with MCP, including exposing public MCP servers and tools.
- Experience designing telemetry for AI systems and using it to guide engineering and product decisions.
- Experience building products with public APIs and evolving data models without breaking external contracts.
- Strong backend engineering skills in Python or TypeScript, including APIs, stateful workflows, data pipelines, and production services.
- Experience collaborating with data scientists, engineers, and product managers and communicating technical trade-offs clearly.
- Bonus: experience with LangGraph, CrewAI, Claude Agent SDK, agentic retrieval, knowledge representation, or Elasticsearch.
Benefits
- Base salary range of $154,000—$243,600 CAD with no variable compensation component; the role is also currently eligible for Elastic’s stock program.
- Company-matched RRSP with dollar-for-dollar matching up to 6% of eligible earnings.
- Distributed work environment with flexible locations and schedules for many roles.
- Health coverage for employees and families in many locations, generous vacation, parental leave of at least 16 weeks, and up to 40 volunteer hours annually.
- Donation matching of up to $2,000 or local currency equivalent.
Tech Stack
Categories
About Elastic
Elastic builds the Elasticsearch-based platform for enterprise search, observability (logging, metrics, APM), and security (SIEM/endpoint). It sells subscriptions and Elastic Cloud managed services with usage-based pricing, combining open source and proprietary features through Elasticsearch and Kibana. Founded in 2012 and headquartered in San Francisco, Elastic is publicly traded on the NYSE and is used by more than half of the Fortune 500.