Principal Software Engineer - Observability & Telemetry Data
Smartsheet3 hours ago
Base Salary
$223k - $258k/yr
Responsibilities
- Architect the end-to-end telemetry data platform, including storage, modeling, querying, retention, partitioning, schema evolution, and separation of storage and compute.
- Define and drive adoption of telemetry semantic conventions, shared identifiers, and schema standards across service teams.
- Lead OpenTelemetry instrumentation migration, including collector architecture, context propagation, and tail-based sampling.
- Architect Databricks and MLflow integrations for AI and agentic telemetry while meeting redaction and governance requirements.
- Instrument Databricks jobs, pipelines, and warehouses with signals for freshness, data quality, lineage, and cost.
- Build telemetry analytics for reliability, incidents, telemetry costs, chargeback, adoption, and coverage.
- Architect high-volume telemetry collection and routing using FluentBit, Kinesis, and OpenTelemetry collectors across multiple regions.
- Own observability cost architecture, including ingest governance, cardinality control, and storage tiering.
- Influence cross-organizational architecture, platform roadmaps, and vendor decisions.
- Lead design and code reviews, author architecture decisions and standards, and mentor engineers.
- Participate in production support and on-call rotations, resolve complex issues, and drive root-cause analysis.
Requirements
- 10+ years of experience building and operating large-scale distributed systems, data platforms, or observability infrastructure, including experience at Principal or Staff level.
- Hands-on production ownership of metrics, logs, and distributed tracing at scale, including Datadog or a comparable backend.
- Deep large-scale data architecture experience with open table formats such as Delta Lake or Apache Iceberg, Spark or comparable distributed processing, streaming ingestion, partitioning, schema evolution, and query performance and cost tuning.
- Practical Databricks experience with jobs, clusters, Unity Catalog, and Delta, plus MLflow experience for model and agent telemetry.
- Practical OpenTelemetry experience with collectors, semantic conventions, context propagation, and sampling strategies including tail-based sampling.
- Experience building high-volume log and telemetry pipelines using FluentBit or Fluentd, Kinesis or Kafka, and search-backend index and mapping design.
- Advanced production AWS and Kubernetes experience, including EKS, ECS Fargate, EC2, Lambda, and CloudWatch.
- 10+ years of programming experience with Go, Java, Python, or Scala, plus strong SQL skills.
- Experience with Terraform and GitOps workflows such as Flux or ArgoCD.
- Demonstrated ability to set technical direction adopted by multiple teams and organizations through architecture decisions, standards, and RFCs.
- Strong incident-response experience and a track record of improving mean time to resolution through better instrumentation and data.
- Bachelor’s degree or equivalent practical industry experience in Computer Science, Engineering, or a related field.
- Legally eligible to work in the United States on an ongoing basis.
- Preferred qualifications include LLM or agentic-system instrumentation, OTel GenAI semantic conventions, data reliability engineering, telemetry cost engineering or FinOps, regulated-environment experience, Prometheus, Grafana, Alertmanager, Snowflake, multi-platform lakehouse or warehouse operations, and Backstage or similar service-catalog or internal developer-platform experience.
Benefits
- Employer-subsidized medical, vision, and dental coverage for full-time employees.
- 401(k) match of 50% of contributions up to the first 6% of eligible pay.
- Monthly stipend supporting work and productivity.
- Flexible Time Away Program and sick time off.
- Employer-sponsored life insurance and short- and long-term disability coverage.
- 12 paid holidays per year.
- Up to 24 weeks of parental leave.
- Personal paid volunteer day.
- Professional growth opportunities, including Udemy online courses.
- Company-funded perks including counseling membership, local retail discounts, and a personal Smartsheet account.
- Full-time role with teleworking available from any registered U.S. location or the Bellevue, Washington office.
Tech Stack
Apache KafkaApache SparkAWSDatabricksDatadogGoGrafanaJavaKubernetesMLflowPrometheusPythonScalaSnowflakeSQLTerraform
Categories
Data EngineeringDevOps
About Smartsheet
Smartsheet is the Intelligent Work Management platform. By aligning people and technology so organizations can move faster and drive innovation, Smartsheet enables its millions of users to achieve more. Visit www.smartsheet.com to learn more. Backed by enterprise-grade security, Smartsheet is used by more than 85% of the companies in the Fortune 500 to implement, manage, and automate processes across a broad array of departments and use cases.