3 days ago
Palo Alto, CA, USASenior
Base Salary
$200k - $260k/yr
Responsibilities
- Build TypeScript services and libraries for high-volume AI model execution, routing, retries, fallbacks, structured outputs, and observability.
- Create evaluation suites, golden datasets, automated graders, LLM-as-judge systems, regression tests, and human-calibrated quality metrics.
- Design and operate experiments and optimization loops for prompts, models, routing policies, agent architectures, context, tool use, retries, caching, parallelism, and workflow policies.
- Establish safe rollout, rollback, champion-challenger, monitoring, and automated gating systems for model and configuration changes.
- Optimize successful task completion per dollar while meeting quality, reliability, and latency requirements.
- Build dashboards and production feedback loops that connect offline evaluations to task, user, business, and customer-segment outcomes.
- Partner with platform, product, and engineering teams to deploy optimization capabilities and translate findings into production improvements.
Requirements
- Strong software engineering skills in TypeScript and Node.js.
- Experience building production systems using LLMs or other machine-learning models and operating high-volume or high-reliability services.
- Experience designing and running rigorous model evaluations, controlled experiments, and experimentation or feedback-loop systems.
- Strong understanding of modern LLM APIs and model behavior, with experience working with production traces, datasets, and experimentation infrastructure.
- Strong SQL and data-analysis skills and the ability to reason quantitatively about quality, latency, reliability, and cost.
- Ability to translate experimental findings into reliable production engineering changes while prompts, models, and execution policies change continuously.
- Python experience for analysis and evaluation tooling is useful but not required.
- Preferred experience includes model routing, agent evaluation, LLM-as-judge systems, online experimentation, bandits, Bayesian optimization, evolutionary search, prompt optimization, reward modeling, fine-tuning, distillation, AI observability, semantic or prefix caching, distributed systems, queues, scheduling, rate limits, provider capacity management, CUDA, GPU kernels, vLLM, or TensorRT-LLM.
Benefits
- Medical, dental, and vision coverage begins on day one.
- Unlimited PTO with a required minimum of one continuous week taken annually.
- 401(k) with company match.
- Remote-flexible work arrangement with office visits when needed.
- Parental leave, professional development training, and employee-led culture initiatives.
- Additional benefits include life and short-term disability insurance, telemedicine, an Employee Assistance Program, Health Savings and Flexible Spending Accounts.
- Base salary is $200,000-$260,000 per year.
Tech Stack
Categories
About Vantaca
Vantaca builds a cloud platform for community association and HOA management companies, combining accounting, payments, workflows, resident communications, and banking integrations with AI-driven automation. It sells subscription software and offers optional managed services for complex financial tasks such as accounting and closings. Founded in 2016 and headquartered in Wilmington, North Carolina, the privately held company focuses exclusively on the community management market across the U.S.