3 months ago
San Francisco, CA, USA or New York, NY, USASenior
Base Salary
$198k - $230k/yr
Responsibilities
- Architect and implement human-in-the-loop and auto-annotation workflows.
- Create ground truth and evaluation criteria for benchmarking models.
- Implement best practices within production pipelines for model compliance.
- Collaborate with Data Science on applied generative decisions.
- Integrate measurement loops into broader infrastructure.
Requirements
- Deep experience in setting up annotation workstreams using tools like Label Studio.
- Experience in implementing model monitoring, versioning, and evaluation loops.
- Strong proficiency in Python for writing scripts and API integrations.
- Understanding of trade-offs between model performance, latency, and cost.
- Comfortable working within AWS and GCP environments.
Benefits
- Work with talented and collaborative colleagues.
- Access to a learning platform for training and tools.
- Surprise meal stipends for remote work.
- 15 days vacation, wellness allowance, and paid parental leave.
- Comprehensive health insurance package and 401k plan.
- Work from home stipend for office setup.