Senior Staff Applied AI Engineer - Context Retrieval
Databricks5 months ago
Base Salary
$229k - $343k/yr
Responsibilities
- Build and own the end-to-end retrieval stack, including query understanding, content understanding, indexing, hybrid retrieval, ranking, and evaluation.
- Retrieve across structured and unstructured assets including tables, columns, SQL queries, dashboards, code, notebooks, jobs, documents, wikis, tickets, chat, images, video, and audio.
- Develop connectors and retrieval adapters for enterprise SaaS systems, accounting for source-specific freshness, permissions, and ranking signals.
- Optimize retrieval for both LLM grounding and human discovery with appropriate quality, explainability, and efficiency signals.
- Build query rewriting, decomposition, intent classification, and entity-resolution capabilities for multi-turn agentic workflows.
- Create pipelines that extract structure, entities, embeddings, summaries, and metadata from supported asset types and keep them current.
- Build search subagents that select sources, plan multi-hop searches, issue follow-up queries, assess retrieval sufficiency, and ground claims in evidence.
- Establish evaluation for retrieval and subagents using ranking metrics, LLM-as-judge harnesses, human labeling, and online experimentation.
- Set the multi-year technical roadmap, make foundational architecture decisions, mentor senior engineers, and partner with Research, Product, and Platform leaders.
Requirements
- 10+ years of software engineering experience, including significant experience building production retrieval, search, or RAG systems at scale.
- Deep information retrieval expertise spanning lexical retrieval, dense retrieval, hybrid retrieval, and learning-to-rank.
- Hands-on experience with RAG architectures, query rewriting, cross-encoder reranking, long-context strategies, and grounding techniques.
- Experience designing agentic systems such as search planners, multi-hop or iterative retrieval, sufficiency checks, and tool-using agents.
- Strong knowledge of relevance evaluation, offline and online experimentation, LLM-as-judge frameworks, and human labeling pipelines.
- Experience indexing and ranking both structured and unstructured data in a unified system.
- Track record of building retrieval systems from zero to one and making foundational architecture decisions.
- Ability to set technical direction, mentor senior engineers, and influence research, product, and platform roadmaps.
- Experience with enterprise SaaS retrieval, permissions, freshness, multi-tenancy, ACL-aware indexing, agentic tool use, or multi-turn retrieval is preferred.
- Open-source IR/search contributions or publications at venues such as SIGIR, KDD, WWW, or EMNLP are preferred.
- Experience training or fine-tuning embedding models, rerankers, or query-understanding models is preferred.
Benefits
- Comprehensive benefits and perks, with region-specific details provided by Databricks.
- Hybrid in-office collaboration is expected in the Mountain View or San Francisco office.
Tech Stack
Categories
About Databricks
Databricks builds a cloud-based data and AI platform centered on the lakehouse architecture, combining data engineering, analytics, and machine learning with Apache Spark, Delta Lake, and MLflow. It sells subscriptions and cloud services to enterprises that need to unify data pipelines and develop large-scale AI and analytics. Founded in 2013 by the creators of Apache Spark and headquartered in San Francisco, the company is privately held and serves organizations across many industries.