Applied AI Engineer
DistantJob2 days ago
Remote, WorldwideSenior
Responsibilities
- Translate scenario specifications, decision models, and assessment rubrics into prompts, application logic, and testable AI behavior.
- Define model integrations with application services and approved data sources, including APIs, tool interactions, schemas, validation, error handling, and permitted actions.
- Design prompts, structured outputs, validation layers, conversational runtimes, controlled personas, session state, and explicit state transitions.
- Implement grounding and retrieval from approved knowledge sources with source references where required.
- Build evaluation assets, run bounded experiments, compare model and retrieval options, and perform regression testing for quality, latency, and cost.
- Take validated prototypes into production with Engineering and monitor behavior after release.
- Develop controls for accreditation compliance, including issue flagging, review steps, disclosure, logging, privacy, human oversight, and AI safety.
- Collaborate with Product, Content Studios, Engineering, and subject-matter experts while explaining technical trade-offs to non-technical colleagues.
Requirements
- Expertise in prompt and context engineering, including system prompts, few-shot examples, prompt chaining, routing, and task decomposition.
- Experience with structured outputs using JSON Schema, function calling, validation, retry, and fallback logic.
- Experience designing retrieval-augmented generation with chunking, metadata filtering, hybrid search, reranking, and citation grounding.
- Knowledge of token budgets, prompt caching, context compaction, long-session memory, agentic systems, state machines, and human-in-the-loop checkpoints.
- Experience defining tool schemas, permission scopes, action limits, evaluation datasets, rubric-based scoring, LLM-as-judge workflows, regression tests, observability, tracing, and LLM security controls.
- Proficiency in Python or TypeScript, API integrations, automated testing, Git, reproducible tests, and CI.
- Hands-on experience building LLM applications with multi-step conversational workflows and collaborating with subject-matter experts.
- Ability to assess privacy, human oversight, AI limitations, and technical trade-offs; experience working with AI coding agents by default.
- Preferred: background in learning products, assessment, simulations, or regulated professional content.
Benefits
- Fully remote work arrangement.
- Long-term opportunity.
Tech Stack
Categories
About DistantJob
DistantJob is a privately held staffing firm that recruits, vets, and places remote software and IT professionals for companies worldwide. Based in Saint-Laurent, Quebec, it supports direct hires and long-term remote engagements and can act as an employer of record to manage local contracts, payroll, and compliance.