
Sr Principal Machine Learning Engineer (Advanced Threat Protection)
Palo Alto Networks2 days ago
Base Salary
$190k - $308k/yr
Responsibilities
- Set organizational technical direction, architectural strategy, engineering roadmaps, and foundation-model adoption plans.
- Lead architectural review boards, establish RFC standards, mentor Staff and Principal engineers, and cultivate engineering rigor.
- Partner with Product Management, security research, data science, infrastructure, and engineering teams to translate business and threat objectives into technical designs.
- Architect fault-tolerant distributed cloud systems processing real-time security telemetry at petabyte scale.
- Design production-grade agentic frameworks with multi-step planning, stateful memory, secure tool execution, and deterministic guardrails.
- Drive ingestion, preprocessing, and feature-engineering pipelines using Apache Spark, SQL, BigQuery, Dataproc, GKE, and GCP SDKs.
- Guide the use and deployment of LLMs, SLMs, and multimodal systems while optimizing cost, latency, privacy, and domain efficacy.
- Diagnose model drift, context dilution, alignment collapse, hallucination spikes, and other production quality failures, then lead fine-tuning and remediation efforts.
- Debug distributed GPU training and inference workloads, including CUDA out-of-memory states, NCCL communication overhead, memory leaks, concurrency locks, and kernel performance bottlenecks.
Requirements
- 10+ years of progressive software engineering and systems design experience delivering high-availability, enterprise-scale platforms.
- Demonstrated ability to author complex system designs, establish data contracts, manage architectural trade-offs, and lead legacy-to-modern migrations.
- Track record of aligning Product, Data, Applied Research, SecOps, and Infrastructure teams around unified technical visions.
- Deep hands-on knowledge of Transformers, attention mechanisms, tokenization, and deployment and serving of LLMs and SLMs.
- Expertise in model diagnosis, data curation, synthetic data generation, supervised fine-tuning, preference alignment, and parameter-efficient tuning.
- Practical experience profiling and debugging distributed GPU workloads and diagnosing memory, concurrency, and cluster communication problems.
- Advanced Python proficiency for AI/ML pipelines and scientific computing.
- Production-level fluency in Go or Java for low-latency concurrent backend microservices.
- Strong data-engineering foundation with Apache Spark, distributed processing, SQL, BigQuery, and GCP orchestration using Dataproc and GKE.
- BS, MS, or PhD in Computer Science, Data Science, Artificial Intelligence, or a related technical discipline.
- Preferred qualifications include cybersecurity expertise in threat modeling, EDR, SIEM, or cloud security posture management; knowledge of L3-L7 protocols and packet telemetry; and open-source, research, or AI-systems community contributions.
Benefits
- Most teams work from the office full time, with flexibility when needed.
- The position offers an annual base salary range of $190,400.00 to $308,000.00, with possible restricted stock units and bonus; compensation figures are excluded from the structured benefits summary.
- Employee benefits information is provided by the employer.
- Reasonable accommodations are available for qualified individuals with disabilities or special needs.
Tech Stack
Categories
About Palo Alto Networks
Palo Alto Networks builds cybersecurity products for enterprises and governments, including next-generation firewalls, cloud security (Prisma Cloud/Prisma Access), and SOC tools (Cortex). It sells subscriptions and appliances with cloud-delivered threat intelligence, securing networks, endpoints, and applications across data centers and public clouds. Founded in 2005 and headquartered in Santa Clara, California, the company is publicly traded on the NYSE under the ticker PANW.