about 4 hours ago
Responsibilities
- Design and build scalable data platforms, data warehouses, and lakehouse architectures.
- Develop and optimize data models, including dimensional modeling and Data Vault principles.
- Utilize advanced SQL for query optimization and large-scale data processing.
- Implement batch, streaming, and CDC-based data ingestion pipelines.
- Work with distributed data processing frameworks like Apache Spark.
- Use Python and/or Scala for data engineering applications.
- Manage workflow orchestration using platforms like Airflow.
Requirements
- 6+ years of experience in Data Engineering.
- Strong expertise in Python, SQL, and Apache Spark.
- Experience building scalable batch and real-time ETL/ELT pipelines.
- Hands-on experience with AWS services including EMR, S3, Glue, and Athena.
- Familiarity with Kafka, Flink, or Kinesis for streaming data processing.
- Strong knowledge of dimensional modeling and data warehousing concepts.
- Experience with modern lakehouse technologies like Delta Lake or Apache Iceberg.
- Expertise in workflow orchestration using Airflow.
- Experience implementing data quality frameworks and monitoring solutions.
Benefits
- Competitive compensation.
- Generous stock options.
- Medical Insurance coverage.
- Opportunity to work with top talent from Silicon Valley.
Tech Stack
Categories
AI & MLData Engineering