Principal AI Platform and Model Operations Engineer - TS/SCI with Polygraph
General Dynamics Information Technology7 days ago
Chantilly, VA, USAStaff+
Base Salary
$162k - $219k/yr
Responsibilities
- Design and operate AI/ML platform services for local model serving, orchestration, registries, experiment and evaluation tracking, retrieval assets, and deployment automation.
- Build governed import and release workflows for models, containers, packages, libraries, prompts, datasets, and vulnerability updates in disconnected environments.
- Implement provenance, signing, SBOMs, malware and vulnerability scanning, license review, approvals, staged validation, and rollback for AI artifacts.
- Deploy and tune inference services, including batching, concurrency, memory, quantization, autoscaling, queueing, resilience, and graceful degradation.
- Define version relationships among models, tokenizers, prompts, retrieval indexes, embedding models, configurations, evaluation sets, applications, and infrastructure.
- Develop GPU, CPU, storage, and network capacity models and advise on hardware, scheduling, consolidation, and cost-performance tradeoffs.
- Implement telemetry for health, latency, throughput, saturation, failures, quality, drift, and usage, integrating it with enterprise observability and ServiceNow.
- Establish backup, recovery, continuity, patching, vulnerability treatment, certificate and secret rotation, and incident procedures.
- Partner with AI Assurance and Security on use-case gates, evaluation thresholds, audits, data restrictions, and release evidence.
- Lead technical reviews, troubleshoot production incidents, mentor engineers, and document reusable platform patterns.
Requirements
- Bachelor’s degree in computer science, Software Engineering, Machine Learning, Information Technology, or a related field, or approved equivalent experience.
- Active TS/SCI clearance with polygraph and U.S. citizenship are required.
- At least 8 years of platform, DevSecOps, SRE, ML engineering, or cloud engineering experience, including at least 3 years supporting production AI/ML platforms.
- Hands-on experience deploying and operating local model-serving platforms such as vLLM, TGI, Triton, TensorRT-LLM, llama.cpp, Ollama, ONNX Runtime, or comparable technologies.
- Strong Linux, container, Kubernetes/OpenShift, networking, storage, GPU, identity, secrets, and infrastructure automation skills.
- Experience with model and artifact registries, experiment tracking, release management, CI/CD, internal package and container repositories, and disconnected dependency management.
- Ability to size inference capacity using model precision, VRAM, context length, concurrency, throughput, latency, and availability requirements.
- Strong Python and automation skills, with experience using Terraform, Ansible, Helm, GitLab CI, Jenkins, or comparable tools.
- Working knowledge of RMF/STIG, software supply-chain security, SBOMs, vulnerability management, logging and audit, and classified data handling.
- Demonstrated technical leadership, incident judgment, cross-team coordination, mentoring, and documentation skills.
- Preferred experience includes PyTorch, Hugging Face, MLflow, Kubeflow, Ray, NVIDIA GPU tooling, CUDA, enterprise AI platforms, A100/H100/L40S accelerators, quantization, optimization, Azure Government, AWS GovCloud, C2S-equivalent environments, or on-premises platforms.
- Preferred credentials include CKA/CKS, Red Hat platform engineering credentials, Microsoft Certified: DevOps Engineer Expert, and relevant Terraform, NVIDIA, or ML-platform certifications.
Benefits
- Comprehensive medical, dental, vision, wellness, disability, life, accident, critical illness, and business travel insurance options are available.
- 401(k) plan with company match and pre-tax and post-tax contribution options.
- Paid vacation, sick, personal, holiday, parental, military, bereavement, and jury-duty leave.
- Full-flex work weeks where possible and broader work-life-balance support.
- Internal mobility support and an AI-powered career tool for identifying career steps and learning opportunities.
- Military-friendly workplace and culture of innovation.
- Onsite position in USA VA Chantilly, with a 40-hour scheduled workweek and no travel required.
Tech Stack
Categories
About General Dynamics Information Technology
General Dynamics Information Technology (GDIT) delivers enterprise IT, systems integration, cybersecurity, cloud, and software engineering services to U.S. federal agencies, including defense and intelligence organizations. It provides contract-based professional and managed services ranging from network operations and unified communications to training and simulation. GDIT is a business unit of General Dynamics (NYSE: GD) and is headquartered in Falls Church, Virginia.