2 days ago
Responsibilities
- Design and implement operators and expressions in a native language for Apache DataFusion and Apache Arrow.
- Extend Apache DataFusion Comet’s Apache Spark integration across Spark versions.
- Accelerate Apache Iceberg and Apache Parquet data reads and writes while preserving Spark-compatible results.
- Profile and benchmark real workloads and tune memory, shuffle, and I/O performance.
- Contribute software and improvements upstream to Apache Spark, Apache DataFusion, Apache Iceberg, and related open-source communities.
- Collaborate with teams running Apache Spark at scale and help roll Comet out across data centers and cloud services.
Requirements
- At least 3 years of professional software engineering experience.
- A proven track record of building distributed systems.
- In-depth knowledge of Apache Spark internals and related data technologies.
- Contributor or committer experience with Apache Spark, Apache DataFusion, or Comet.
- Experience with columnar or vectorized query execution, performance profiling, or query-engine optimization.
- Good knowledge of Apache Iceberg and Apache Parquet internals.
- Experience working on or building Apache Spark connectors to external data sources.
- Knowledge of Kubernetes or other compute platforms.
- BS, MS, or PhD in Computer Science or a related field, or equivalent practical experience.
Tech Stack
Categories
BackendData Engineering
About Apple
Apple designs and sells consumer electronics, software, and services for consumers and professionals worldwide, including iPhone, Mac, iPad, Apple Watch, and AirPods, plus platforms like iOS/macOS and services such as the App Store, iCloud, Music, and TV+. Its business combines device sales with services and subscriptions and in-house silicon design. Founded in 1976, Apple is headquartered in Cupertino, California, and trades on NASDAQ as AAPL.