Senior AI Engineer — Inference & Agent Systems
Arcana Analytics5 months ago
Responsibilities
- Optimize inference latency, including achieving sub-400ms time to first token, streaming partial output, KV caching, prompt compression, and dynamic context management.
- Build multi-provider model routing across OpenAI, Anthropic, Gemini, and open-weight models based on latency, cost, and task type.
- Design and implement parallel Plan-Execute-Synthesize agent pipelines using DAGs and build reliable Temporal orchestration with retries, timeouts, recovery, and idempotency.
- Implement structured output validation, retry handling for malformed LLM output, graceful degradation, and reliable tool-call schemas across providers.
- Own the evaluation harness, including ground-truth datasets, automated scoring, LLM-as-judge pipelines, regression detection, latency testing, and adversarial test cases.
- Optimize model serving, cold starts, and asynchronous workers while providing observability across tokens, tool calls, and synthesis steps.
Requirements
- Production experience shipping systems at meaningful scale with a deep understanding of performance and reliability.
- Strong experience with inference pipelines, multi-step agent systems, evaluation harnesses, streaming LLM responses, or resilient handling of LLM non-determinism.
- Experience with Go, Python, Temporal, Kafka, PostgreSQL, and Docker is valuable, though depth matters more than exact stack matching.
- Experience with real-time AI products, model-serving infrastructure, agent platforms, or trusted evaluation infrastructure is sought.
- Fine-tuning models, using LangChain or LlamaIndex, or having a strong ML research background without systems exposure are considered weaker but acceptable signals.
Benefits
- Remote work in India (BLR/Remote India)
- High ownership on a small team with decisions shipping to production
- No cover letter required
Tech Stack
Categories
About Arcana Analytics
Arcana enables institutional investors to understand their portfolio risks, decompose single stock & book performance, drill into crowding, and isolate their idiosyncratic differentiation. Built on our proprietary crowding, ownership, factor risk, and performance datasets. The company's investors include D1 Capital, Duquesne (Stan Druckenmiller), Tiger Global, Abstract Ventures, GoldenTree Asset Management, Ryan Roslansky (CEO LinkedIn), and Akshay Kothari (COO Notion), among others.