Sr. Applied AI Engineer
O.C. Tanner2 months ago
Salt Lake City, UT, USASenior
Responsibilities
- Design, build, deploy, and support production-grade agentic AI systems with explicit goals, policies, constraints, and guardrails.
- Build orchestration patterns for multi-step workflows, tool calling, MCP servers, state management, memory, retries, recovery, and human-in-the-loop controls.
- Develop user-centered AI interactions with conversational flows, feedback loops, confidence handling, explainability, escalation paths, and graceful failure modes.
- Develop and operate RAG systems with ingestion, chunking, embeddings, vector and hybrid retrieval, reranking, evaluation, and citation or traceability strategies.
- Define evaluation frameworks using offline test sets, regression suites, adversarial testing, groundedness and faithfulness scoring, task-completion metrics, and production monitoring.
- Instrument systems for observability across model calls, prompts, tools, decisions, retrieved context, latency, cost, errors, and user feedback.
- Implement responsible AI safeguards including prompt-injection defense, data access controls, PII protection, bias and toxicity detection, misuse prevention, audit logging, and policy enforcement.
- Optimize model selection, prompts, context windows, caching, routing, inference, latency, throughput, reliability, and cost.
- Mentor engineers on applied AI, agent design, RAG, evaluation, safety, observability, and production support.
Requirements
- At least 5 years of software engineering experience with strong Python proficiency.
- At least 2 years of experience building production ML or agentic AI systems.
- At least 1 year of hands-on experience with agentic frameworks such as LangGraph, CrewAI, or AutoGen, or equivalent.
- Experience building production AI systems with agents, MCP servers, multi-step reasoning, and multi-turn conversation.
- Experience deploying RAG systems with embedding models, vector databases, hybrid search, and retrieval optimization.
- Experience designing LLM strategies involving tool calling, structured outputs, prompt engineering, and context-window management.
- Experience implementing AI safety and evaluation pipelines covering bias detection, PII leakage, faithfulness scoring, toxicity, and prompt-injection mitigation.
- Experience optimizing models for inference efficiency, latency, and cost management.
- A bachelor's degree in Computer Science, Machine Learning, or a related field is strongly preferred.
- AWS Certified Machine Learning Engineer – Associate certification or equivalent is strongly preferred.
- Cloud AI infrastructure management using AWS services and Terraform is strongly preferred.
- AI observability experience with OpenTelemetry, Langfuse, or equivalent is strongly preferred.
Tech Stack
Categories