3 months ago
Responsibilities
- Maintain large-scale pipelines for processing web corpora
- Develop filtering and quality-scoring systems to identify high-value web documents
- Analyze web-data composition across domains, languages, and time periods
- Develop and maintain high-performance deduplication pipelines
- Collaborate with researchers and engineers to ensure data pipelines meet the needs of advanced language models
Requirements
- Strong software engineering skills with proficiency in Python and experience building data pipelines
- Familiarity with data-processing frameworks such as Apache Spark, Apache Beam, Pandas, or similar tools
- Experience working with large-scale web datasets
- Knowledge of data-quality assessment techniques and experimentation with data mixtures
- Interest in bridging research and engineering to solve complex data challenges in AI model training
- A paper at a top-tier venue such as NeurIPS, ICML, ICLR, AIStats, MLSys, JMLR, AAAI, Nature, COLING, ACL, or EMNLP is a bonus
Benefits
- Open and inclusive culture and work environment
- Weekly lunch stipend, in-office lunches, and snacks
- Full health and dental benefits, including a separate mental health budget
- 100% parental leave top-up for up to 6 months
- Personal enrichment benefits for arts and culture, fitness and well-being, quality time, and workspace improvement
- Remote-flexible work with offices in Toronto, New York, San Francisco, London, and Paris
- Co-working stipend
- Six weeks of vacation, equivalent to 30 working days
Tech Stack
Categories
Data Engineering
About Cohere
Cohere is the leading security-first enterprise AI company. We build cutting-edge foundation models and end-to-end AI products designed to solve real-world business problems. We partner closely with companies to deliver seamless integration, full customization, and easy-to-use solutions for their workforce and customers. Our all-in-one platform offers enterprises the highest levels of data security, privacy and optionality to deploy across all major cloud providers, private cloud environments, or on-premises. HQ: 171 John Street, 2nd Floor, Toronto, ON M5T 1X3