Senior Software Engineer – Network Observability
Clockwork.io1 day ago
Base Salary
$140k - $210k/yr
Responsibilities
- Design, develop, and scale high-performance network monitoring platforms for RDMA, RoCE, InfiniBand, and TCP/IP infrastructure.
- Build backend telemetry services, observability dashboards, alerts, diagnostics, anomaly detection, SLA monitoring, and traffic-analysis workflows.
- Troubleshoot production issues across application, operating-system, server, RDMA, and network layers while optimizing low-latency collection, aggregation, and alerting.
- Collaborate cross-functionally to deliver scalable solutions, improve engineering practices, automate operational workflows, and guide the platform’s technical direction.
Requirements
- Bachelor’s or master’s degree in computer science, computer engineering, electrical engineering, or a related technical field.
- Strong hands-on programming experience in C++, Go, Python, Rust, or similar systems programming languages.
- Experience delivering complex infrastructure projects and designing and implementing distributed systems.
- Experience building distributed systems, backend services, telemetry pipelines, or observability platforms.
- Hands-on experience with RDMA, RoCE, InfiniBand, or other high-performance network fabrics.
- Knowledge of libibverbs, RDMA verbs, RDMA CM, queue pairs, completion queues, memory registration, and related RDMA concepts.
- Strong knowledge of Linux networking, TCP/IP, DNS, HTTP, routing, MTU, congestion control, packet loss, and performance tuning.
- Experience with traceroute-style diagnostics, path discovery, network reachability checks, synthetic probes, or active network measurements.
- Experience with monitoring and visualization platforms such as Prometheus, Grafana, Datadog, Splunk, or OpenTelemetry.
- Strong debugging skills across software, operating-system, server, and network layers.
- Experience operating production systems in Linux-based environments.
- Strong technical judgment and ability to build reliable, scalable, and maintainable systems.
- Preferred experience supporting AI/ML, HPC, storage, or GPU cluster infrastructure workloads.
- Preferred experience with large-scale RoCE or InfiniBand deployments, NCCL, distributed training infrastructure, or AI cluster diagnostics.
- Preferred experience with eBPF, XDP, DPDK, perf, tcpdump, Wireshark, ethtool, iproute2, rdma-core, or Linux kernel networking tools.
- Preferred experience with AWS, GCP, Azure, Kubernetes, service discovery, configuration management, infrastructure automation, time-series data systems, alerting pipelines, or high-cardinality telemetry platforms.
Benefits
- Competitive compensation and a great benefits package.
- Catered lunch.
- Friendly and inclusive workplace culture.
- Eligible to participate in the company’s equity program, which may include stock options subject to the equity plan, approval, and vesting schedules.
- Expected base salary range of $140,000–$210,000 for candidates hired at the posted level.
Tech Stack
Categories
About Clockwork.io
Clockwork.io pioneers Software-Driven AI Fabrics™, delivering a programmable software layer that makes large-scale AI clusters observable, deterministic, and resilient by design to drive continuous workload progress and peak cluster utilization. Its FleetIQ platform enables enterprises to train, deploy, and serve the world's most demanding AI workloads faster, more reliably, and at lower cost. Companies including Uber, Wells Fargo, DCAI, Nebius, Nscale, and White Fiber trust Clockwork.io to power their AI infrastructure. Learn more at www.clockwork.io