29 days ago
Palo Alto, CA, USAStaff+
Responsibilities
- Define the model capabilities and behaviors that should be measured.
- Build and ship evaluation pipelines with statistical analysis and reporting.
- Partner with model training and research teams to embed evaluation into development workflows.
- Prototype user studies and behavioral experiments based on real-world model use.
- Develop quantitative metrics for subjective qualities such as understanding, naturalness, and adaptability.
- Write the pipelines and tooling required to implement evaluation systems.
Requirements
- Experience designing or implementing evaluation frameworks for generative audio, text, or multimodal models.
- Strong technical and analytical skills with the ability to develop production-ready systems from open-ended research ideas.
- Ability to create novel quantitative metrics for inherently subjective qualities.
- Hands-on engineering experience building evaluation systems and tooling.
- AI modeling experience, including training, fine-tuning, or shipping models, is preferred.
- Background in audio modeling such as Speech-to-Text or Text-to-Speech is preferred.
- Multilingual experience, particularly evaluating semantic nuance across languages, is preferred.
Categories
About Sanas.ai
Sanas builds real-time speech AI that translates accents and enhances clarity on live conversations, helping multilingual agents be more easily understood. It sells the technology as SaaS via APIs, call-center integrations, and an agent desktop app to contact centers, BPOs, and enterprise support and sales teams. The company is privately held and headquartered in Palo Alto, California.