16 days ago
Bengaluru, IndiaStaff+
Responsibilities
- Architect and manage Apache Iceberg lakehouse tables, including schema evolution, partitioning, snapshots, compaction, and metadata optimization.
- Deploy, operate, and performance-tune distributed analytical query and data warehouse clusters.
- Optimize Parquet storage layouts, compression, encoding, row groups, and predicate pushdown.
- Build reliable batch and streaming ETL/ELT pipelines with data quality, idempotency, backfills, and observability.
- Design KEDA-based event-driven autoscaling for data workloads on Kubernetes.
- Provision and maintain cloud and Kubernetes infrastructure with Terraform, Helm, and GitOps workflows.
- Diagnose and tune self-hosted open-source warehouse performance across compute, memory, storage, and networking.
- Define SLOs, build monitoring and alerting, participate in on-call, and lead incident retrospectives.
- Review designs and code, mentor engineers, and drive technical direction and platform initiatives end to end.
Requirements
- At least 5 years of experience designing, building, and operating large-scale data platforms, lakehouse architectures, and distributed analytical systems.
- Hands-on expertise with Apache Iceberg or Hudi and Parquet, including partitioning, schema evolution, compaction, snapshot management, and storage optimization.
- Experience operating and performance-tuning analytical query engines such as StarRocks, ClickHouse, or Trino.
- Proficiency building reliable batch and streaming pipelines with Apache Spark and Flink, including strong data quality, idempotency, fault tolerance, and observability practices.
- Experience operating production-grade, multi-region cloud platforms across AWS, Azure, or OCI.
- Experience mentoring engineers and leading platform initiatives with a focus on scalability, reliability, performance, and cost efficiency.
Tech Stack
Amazon RedshiftApache FlinkApache KafkaApache SparkAWSAzureClickHouseDatabricksGrafanaHelmKubernetesOracle CloudPrometheusTerraform
Categories
Data EngineeringDevOps
About Eightfold
Eightfold AI provides a subscription Talent Intelligence Platform that uses machine learning to support talent acquisition, internal mobility, and workforce upskilling for large organizations. Its software integrates with common ATS and HRIS systems and is deployed globally in more than 155 countries and 24 languages. Founded in 2016 and headquartered in Santa Clara, California, Eightfold is a privately held company.