Cisco

Principal Software Engineer, Applied AI – Splunk Observability Cloud

Cisco
Apply
3 days ago

Base Salary

$241k - $417k/yr

Responsibilities

  • Define the technical trajectory for agentic investigation capabilities in Splunk Observability Cloud.
  • Architect evaluation systems and retrieval layers for AI reasoning over telemetry and other observability data.
  • Establish production standards for AI agent latency, cost, security, and safe autonomous remediation.
  • Develop evaluation frameworks covering dataset creation, offline and online scoring, and release validation.
  • Set long-term technical direction while bridging AI research and mission-critical product features.
  • Mentor engineers, lead design reviews, create technical documentation, and partner with chief architects.
  • Use customer-zero or field-deployed feedback to drive product improvements.

Requirements

  • 12+ years of professional software engineering experience.
  • 3+ years designing, building, and deploying production-grade LLM-based systems, including at least one agentic system with active tool use.
  • Experience developing AI/ML evaluation frameworks with dataset creation, offline and online scoring, and release validation.
  • Experience engineering retrieval and context pipelines for structured and semi-structured data such as telemetry, logs, or time-series databases.
  • Knowledge of latency optimization, cost management, multi-tenancy, and security guardrails for production AI systems.
  • Preferred experience with observability, SRE, incident management, distributed tracing, and alerting workflows.
  • Preferred experience with cloud-native infrastructure at scale, specifically Kubernetes.
  • Preferred active open-source contributions in AI, agent frameworks, or observability ecosystems such as OpenTelemetry or MCP.
  • Preferred experience leading customer-zero or field-deployed programs.
  • Strong communication, technical-direction, mentoring, design-review, and technical-documentation experience.

Benefits

  • Medical, dental, and vision insurance; 401(k) with Cisco matching contribution; paid parental leave; disability coverage; and basic life insurance, subject to plan eligibility.
  • Potential eligibility for Cisco restricted stock unit grants and annual bonuses for non-sales roles.
  • Paid holidays, floating holiday, birthday leave, year-end shutdown, personal wellness days, vacation or flexible time off, sick time, and family emergency leave according to applicable policies.
  • Optional 10 paid volunteer days per full calendar year.
  • Salary range applies to new hires in U.S. and/or Canada locations, with location-specific ranges provided for certain states.

Tech Stack

Categories

Cisco

About Cisco

10,000+ employees

Cisco designs and sells networking, security, and collaboration platforms for enterprises, service providers, and governments, spanning routers and switches, Wi‑Fi, firewalls, zero‑trust, observability, and cloud-managed IT (Meraki) plus Webex. Its business model mixes hardware, software subscriptions, and support/consulting services. Founded in 1984 and headquartered in San Jose, California, Cisco is a public company traded on Nasdaq and serves customers across data centers, campuses, and service provider networks.

Contact me