3 months ago
Responsibilities
- Design and scale distributed pipelines for preprocessing, dataset generation, and repeated dataset refreshes.
- Own workflow orchestration, job scheduling, monitoring, and failure recovery for large-scale data-processing jobs.
- Implement and maintain containerized pipeline infrastructure using Kubernetes or equivalent orchestration systems.
- Optimize cloud-based data storage and movement across AWS, GCS, and Azure for cost, throughput, and operational efficiency.
- Define standards for dataset storage layout, versioning, caching, retention, and access patterns.
- Build curation pipelines for selecting, filtering, and retaining video and image content, including image-text pair datasets.
- Build VLM-based captioning and metadata-generation workflows across video and image data.
- Develop quality and aesthetic scoring models, CLIP-based semantic filtering, and other signal-extraction approaches for data selection.
- Build large-scale near-deduplication and exact-deduplication tooling for video corpora.
- Analyze dataset composition, identify quality issues, and improve curation logic and standards for training-ready video data.
Requirements
- Strong hands-on experience building or scaling large-scale machine-learning data systems and pipelines, including dataset curation, filtering, and quality improvement.
- Experience with distributed data processing frameworks such as PySpark or Ray and orchestration tools such as Airflow or equivalent.
- Familiarity with Docker, Kubernetes, and container orchestration.
- Experience with AWS, GCS, and/or Azure for cloud-based data storage and compute, including cost, throughput, storage layout, and access-pattern tradeoffs.
- Experience with VLM-based captioning pipelines or quality and aesthetic scoring models for video or image data, including image-text pair dataset curation.
- Familiarity with CLIP-based or embedding-based filtering and semantic data selection techniques.
- Familiarity with FFmpeg, PyAV, DALI, OpenCV, Decord, torchvision, PyTorchVideo, or torchaudio.
- Proficiency in Python.
- Strong problem-solving, communication, and documentation skills.
Benefits
- Competitive salary and generous company equity
- Personal time off and paid holidays
- Health insurance
- Global travel insurance for international travel
- Monthly spending stipend of $500 (~S$635)
- Equipment provided for a home office
Tech Stack
Categories
Data EngineeringML Engineering
About Cantina
Cantina Labs is a social AI company, developing a suite of advanced real-time models that push the boundaries of expression, personality, and realism. We bring characters to life, transforming how people tell stories, connect, and create. We build and power ecosystems. Cantina, our flagship social AI platform, is just the beginning.