Software Engineer
Spellbrushalmost 2 years ago
Tokyo, Japan or San Francisco, CA, USAMid Level
Responsibilities
- Define processes and infrastructure that transform and make data available across the company.
- Design, build, and monitor pipelines for embeddings, video, bounding boxes, and other data.
- Partner with AI researchers to gather requirements and deliver scalable technical solutions.
- Build and operate data pipelines across distributed systems, cloud platforms, and multiple data stores.
- Ingest values and embeddings from state-of-the-art AI models and support large-scale data processing.
Requirements
- At least 3 years of software engineering experience with a broad generalist skill set.
- Experience with distributed systems and administration of big data tools such as Ray, Spark, or Airflow.
- Hands-on experience shipping scalable cloud data solutions using AWS, GCP, or Azure.
- Experience working across data stores such as Snowflake, Redshift, Hive, SQL, and NoSQL systems.
- Ability to extract and transfer very large datasets, including complex SQL queries and multi-petabyte data transfers.
- Comfort with scripting in TypeScript, Python, or Bash.
- Experience designing and building highly scalable and reliable data pipelines.
- Ability to work directly with AI researchers and operate effectively on small, fast-paced teams.
- Interest in anime and the anime aesthetic is preferred.
Benefits
- On-site collaboration in Tokyo or San Francisco.
- Visa sponsorships are available.
Tech Stack
Amazon RedshiftApache AirflowApache HiveApache SparkAWSAzureBashGoogle BigQueryGoogle Cloud PlatformPythonSnowflakeSQLTypeScript
Categories
BackendData Engineering
About Spellbrush
Spellbrush builds anime-focused video games and trains generative image models for artists and game production. The company is the team behind niji・journey, the anime image generator available through Midjourney. Founded in 2018 and headquartered in San Francisco with a research office in Tokyo, it operates its own GPU cluster to train models and develops in-house games with an emphasis on turn-based tactics.