Principal Machine Learning Engineer
Palo Alto Networks3 days ago
Base Salary
$163k - $264k/yr
Responsibilities
- Provide technical leadership for the architecture and development of AI Canvas AI capabilities from design through production and continuous improvement.
- Build scalable systems using LLMs, RAG, AI agents, assistants, tools, skills, and agent harnesses.
- Develop natural-language-to-query workflows supporting investigation, clarification, troubleshooting, and follow-up interactions.
- Design agentic AI architecture including orchestration, context management, tool invocation, memory, reasoning workflows, and multi-step execution.
- Build evaluation frameworks using automated metrics, LLM-based evaluators, human evaluation, and production datasets.
- Implement guardrails and safety mechanisms to reduce hallucinations, enforce constraints, and improve responsible AI behavior.
- Improve model and system quality through prompt engineering, retrieval strategies, model selection, fine-tuning, and experimentation.
- Build scalable knowledge-retrieval systems grounded in complex security datasets.
- Partner with backend and platform engineers on production APIs, services, and cloud infrastructure for AI workloads.
- Establish practices for observability, debugging, tracing, versioning, reproducibility, testing, and monitoring of LLM and agent systems.
- Evaluate emerging models, frameworks, and AI infrastructure for product adoption.
- Mentor engineers and influence technical direction across teams.
Requirements
- Bachelor’s degree in Computer Science, Machine Learning, Engineering, or a related technical field, or equivalent practical experience.
- At least 7 years of software engineering, machine learning engineering, or related industry experience, including significant production-system experience.
- Strong experience building machine learning or AI-powered applications at scale.
- Hands-on experience building applications with large language models and generative AI technologies.
- Experience with RAG, AI agents, AI assistants, tool-calling systems, agent orchestration, or LLM-based workflows.
- Strong Python proficiency and experience developing production-grade backend or ML services.
- Experience designing AI/ML evaluation systems, including offline evaluation, regression testing, quality measurement, and production monitoring.
- Understanding of embeddings, retrieval, ranking, prompt engineering, model inference, and experimentation.
- Experience building scalable systems on GCP, AWS, or Azure.
- Strong software engineering fundamentals in system design, distributed systems, APIs, testing, CI/CD, and observability.
- Ability to lead complex cross-team technical initiatives while remaining hands-on.
- Strong communication skills and ability to translate ambiguous product problems into technical architectures and execution plans.
- Preferred: Master’s or PhD in Computer Science, Machine Learning, Artificial Intelligence, or a related technical field.
- Preferred: Experience with LLM-powered products, agentic AI platforms, AI development frameworks, evaluation harnesses, vector databases, search and retrieval infrastructure, Docker, Kubernetes, highly available AI inference services, AI security, or cybersecurity.
Benefits
- Most teams work from the office full time, with flexibility when needed.
- The role is eligible for immigration sponsorship.
- Benefits may include restricted stock units and a bonus in addition to base compensation.
Categories
About Palo Alto Networks
Palo Alto Networks builds cybersecurity products for enterprises and governments, including next-generation firewalls, cloud security (Prisma Cloud/Prisma Access), and SOC tools (Cortex). It sells subscriptions and appliances with cloud-delivered threat intelligence, securing networks, endpoints, and applications across data centers and public clouds. Founded in 2005 and headquartered in Santa Clara, California, the company is publicly traded on the NYSE under the ticker PANW.