5 months ago
Responsibilities
- Maintain large-scale pipelines for processing web corpora
- Develop filtering and quality-scoring systems to identify high-value web documents
- Analyze web-data composition across domains, languages, and time periods
- Develop and maintain high-performance deduplication pipelines
- Collaborate with researchers and engineers to ensure data pipelines meet the needs of advanced language models
Requirements
- Strong software engineering skills with proficiency in Python and experience building data pipelines
- Familiarity with data-processing frameworks such as Apache Spark, Apache Beam, Pandas, or similar tools
- Experience working with large-scale web datasets
- Knowledge of data-quality assessment techniques and experimentation with data mixtures
- Interest in bridging research and engineering to solve complex data challenges in AI model training
- A paper at a top-tier venue such as NeurIPS, ICML, ICLR, AIStats, MLSys, JMLR, AAAI, Nature, COLING, ACL, or EMNLP is a bonus
Benefits
- Open and inclusive culture and work environment
- Weekly lunch stipend, in-office lunches, and snacks
- Full health and dental benefits, including a separate mental health budget
- 100% parental leave top-up for up to 6 months
- Personal enrichment benefits for arts and culture, fitness and well-being, quality time, and workspace improvement
- Remote-flexible work with offices in Toronto, New York, San Francisco, London, and Paris
- Co-working stipend
- Six weeks of vacation, equivalent to 30 working days
Tech Stack
Categories
Data Engineering
About Cohere
Cohere builds large language models and an enterprise AI platform that companies use for search, summarization, and workflow automation, delivered via API or private deployments. Founded in 2019 and headquartered in Toronto, it focuses on multilingual models, data controls, and options to run across major clouds or on-premises. The business is privately held and serves security- and compliance-sensitive organizations.