3 months ago
Prague, CzechiaSenior / Staff+
Responsibilities
- Set up and maintain LLM observability frameworks and tools.
- Improve data annotation tooling.
- Ensure the stability of LLM calls through rate limits, provisioned throughput, backups, and related measures.
- Help drive security review processes for AI vendors and providers.
- Recommend LLM cost optimizations such as caching, batching, and identifying costly workflow components.
- Host fine-tuned and open-weight machine learning models.
- Help develop LLM evaluation tooling and frameworks, with a current focus on agentic evaluations.
- Build platform tooling that enables non-technical users such as product managers to iterate on prompts.
Requirements
- 5+ years of experience building and operating software systems end-to-end.
- Hands-on experience with ML infrastructure, including model serving, training pipelines, or production LLM integrations.
- Strong understanding of cloud infrastructure and distributed systems, primarily AWS.
- Familiarity with observability tooling and cost management for LLM workloads.
- Experience with or openness to Python, Kubernetes, and Terraform.
- Clear communication, high ownership, and comfort working in a remote-first, asynchronous environment.
- Experience with evaluation frameworks, annotation tooling, or prompt management platforms is a bonus.
Benefits
- Remote-first, asynchronous work environment.
- Independent contractor arrangement; contractors are responsible for their own taxes and legal compliance.
- Open only to candidates residing in or willing to relocate to Czechia or Slovakia who have the legal right to work there.
Tech Stack
Categories
