10 hours ago
Bengaluru, IndiaSenior
Responsibilities
- Build and optimize the next-generation lakehouse platform.
- Enhance Apache Spark stability, usability, and performance, including its internals and execution behavior.
- Extend Apache Iceberg capabilities and build internal storage and metadata services.
- Troubleshoot and resolve production issues while minimizing downtime.
- Maintain and improve system monitoring, logging, and alerting.
- Deliver production-quality software using AI coding tools and provide agents with appropriate context, specifications, constraints, and validation criteria.
- Capture business, architecture, codebase, and operational knowledge so AI coding agents can execute complex tasks accurately and autonomously.
Requirements
- At least 8 years of validated, successful software engineering experience.
- Strong programming skills in Java or Scala.
- Comprehensive knowledge of Apache Spark internals, including the Catalyst optimizer, Tungsten execution engine, shuffle mechanics, AQE, and memory management.
- Solid understanding of Apache Parquet and table formats such as Apache Iceberg, Delta, and Apache Hudi, including snapshots, manifests, partition specifications, and table metadata.
- Familiarity with Linux/Unix systems, shell scripting, and system commands.
- Understanding of networking principles and distributed environments.
- Strong analytical and problem-solving skills for defect, performance, and distributed-systems issues.
- Experience building production-grade software with AI coding agents while maintaining quality, availability, and maintainability.
- Hands-on experience delivering complex production-quality software using AI coding tools such as Claude Code or Codex.
- Ability to work effectively with global teams.
- Preferred: contributions to Apache Spark or Apache Iceberg open-source projects.
- Preferred: experience designing and implementing distributed systems.
- Preferred: familiarity with HDFS, HBase, and YARN.
Tech Stack
Categories
BackendData Engineering
About TCGplayer
TCGplayer operates an online marketplace and software tools for buying and selling trading card games and related collectibles, serving hobby shops and individual sellers. It generates revenue through marketplace fees, seller subscriptions, and fulfillment programs such as TCGplayer Direct. An eBay subsidiary, the company provides authentication and logistics services that help stores list inventory at scale and reach buyers across the U.S. and internationally.