Software Engineer, Analytics Platform
OpenAI
about 1 year ago
San Francisco, CA, USA
Mid Level / Senior
Base Salary
$295k - $445k/yr
Responsibilities
- Build and maintain large-scale stream and batch processing pipelines using Kafka, Spark, Flink, and Trino/Presto.
- Develop a general-purpose data processing platform for handling massive datasets.
- Scale applications for ML research to ensure smooth operation as workloads grow.
- Ensure the security, integrity, and compliance of data according to industry standards.
- Empower engineers and researchers with excellent data tooling and systems.
- Partner with product engineers and other teams to build technical foundations.
- Participate in an on-call rotation to respond to critical incidents.
Requirements
- Proficient in Python and backend development with experience in large codebases.
- Experience building and operating large-scale stream and batch processing pipelines.
- Hands-on experience with Kubernetes, Terraform, and production system deployment.
- Knowledge of access control, provenance, auditing, and large-scale data movement.
- Passion for building systems that provide key insights in ML training workflows.
- Comfortable in a fast-moving environment and making trade-offs for impact.
- Understanding of data transformations in ML training and inference workflows is a plus.
Benefits
- Flexible remote work options within the US.
- Hybrid work model of 3 days in the office per week.
- Relocation assistance for new employees.
Tech Stack
Apache FlinkApache KafkaApache SparkKubernetesPrestoPythonTerraform
Categories
AI & MLBackendData Engineering