3 days ago
Base Salary
$112k - $179k/yr
Responsibilities
- Monitor production AI services and agent workflows, including health, latency, errors, model and token usage, costs, tool-call failures, and workflow completion.
- Integrate AI and agent telemetry into observability dashboards, monitoring, and alerting systems.
- Investigate abnormal or failed workflows and coordinate remediation with software, platform, AI engineering, and SRE teams.
- Maintain runbooks, escalation procedures, troubleshooting guidance, recovery processes, and operational evidence.
- Develop repeatable evaluation scenarios and assess task completion, output quality, error patterns, tool use, and operational suitability.
- Coordinate regression testing for changes to models, prompts, tools, workflow logic, and retrieval sources.
- Maintain version control and change records for prompts, agent instructions, tools, models, evaluation datasets, and workflow configurations.
- Coordinate approved changes, configuration validation, rollback, access controls, auditability, and safeguards for production AI services.
- Report service health, model usage, operating costs, evaluation results, unresolved issues, and operational improvement opportunities.
Requirements
- Active TS/SCI clearance with CI Polygraph is required.
- BS/BA and 5 years of experience are required, with an additional 4 years of experience potentially accepted in lieu of a degree.
- Approximately 5–8 years of experience in platform operations, site reliability engineering, cloud engineering, software operations, applied AI/ML operations, or production application support.
- Experience operating and troubleshooting production applications or cloud services, including monitoring, logging, metrics, incident response, and configuration or change management.
- Experience with repeatable test cases, regression testing, quality measures, or operational performance analysis.
- Working knowledge of foundation-model or LLM applications, including prompts, model endpoints, tool calls, and retrieval-augmented applications.
- Experience with AWS or comparable cloud platforms.
- Preferred experience includes AWS Bedrock, LLM or agent evaluation frameworks, OpenTelemetry, Amazon CloudWatch, OpenSearch, Prometheus, Grafana, ServiceNow, GitLab, Kubernetes, containerized environments, cloud and model cost monitoring, risk management, continuous monitoring, security evidence collection, and classified or regulated environments.
- Strong analytical, troubleshooting, documentation, and communication skills are required.
Benefits
- Fully onsite in Chantilly, Virginia.
- Target base salary range is $112,000–$179,000, with potential eligibility for overtime, shift differential, and discretionary bonus.
Tech Stack
About Peraton
Peraton builds and integrates mission systems, cyber, space, and intelligence solutions for U.S. defense, intelligence, and civil agencies. It delivers enterprise IT, satellite and terrestrial communications, and spectrum management under government contracts. Founded in 2017, the company is headquartered in Reston, Virginia and is privately held by Veritas Capital.