9 hours ago
Responsibilities
- Own AI-powered feature behavior across product surfaces, including quality, safety, variance, and failure modes.
- Design prompting, retrieval, routing, and tool-use strategies for non-deterministic systems.
- Build and maintain evaluation systems using statistical signals, distributions, trends, golden datasets, regression detection, and human feedback.
- Detect and resolve hallucinations, partial correctness, instruction drift, and context sensitivity.
- Implement guardrails, fallbacks, and degradation paths for unexpected model behavior.
- Partner with product and infrastructure teams on quality, cost, latency, robustness, and shipping decisions.
- Influence model selection, model behavior, and tool design for real user workflows.
- Set quality standards and mentor engineers on non-deterministic system design and evaluation.
Requirements
- 6+ years of professional engineering experience with ownership of complex production systems.
- Experience owning AI/LLM behavior beyond basic integration, including variance and failure-mode mitigation.
- Ability to reason about probabilistic systems and quality, cost, recall, precision, speed, and robustness tradeoffs.
- Experience designing or maintaining evaluation frameworks, golden datasets, regression detection, or human-in-the-loop feedback loops.
- Strong product intuition and ability to define quality for non-deterministic outputs.
- Ability to work independently in ambiguous problem spaces and establish standards others rely on.
- Bonus: production experience with RAG, agentic systems, or tool-using models.
- Bonus: familiarity with vector databases, embeddings, or retrieval pipelines.
- Bonus: exposure to fine-tuning, model routing, or post-training techniques.
- Bonus: experience building shared AI infrastructure or mentoring engineers.
Tech Stack
Categories
About Retool
Retool builds a developer platform for quickly creating internal apps, workflows, and automations by combining visual components with code and connecting to existing databases and APIs. It sells cloud and self‑hosted enterprise subscriptions with governance, security, and permissions features for large organizations. Founded in 2017 and headquartered in San Francisco, it is privately held and used by engineering and operations teams across many industries.