
Data Engineer
Sembcorp Group20 hours ago
Singapore, SingaporeSenior
Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines using Databricks, PySpark, Python, SQL, and Delta Lake.
- Build and support data solutions based on Lakehouse and medallion architecture principles.
- Integrate data from databases, APIs, files, streaming sources, and IoT sources.
- Implement data quality, reconciliation, incremental processing, schema management, and recovery mechanisms.
- Monitor and troubleshoot production pipelines, data freshness, performance, and reliability issues.
- Identify repetitive BAU activities and develop automation or agentic solutions to reduce manual effort.
- Explore Databricks and AI capabilities to improve engineering and operational workflows.
- Apply software engineering and DevOps practices including testing, logging, version control, and CI/CD.
- Participate in technical discussions, propose practical solutions, share knowledge, and experiment with new approaches.
Requirements
- At least 5 years of hands-on data engineering experience.
- Strong hands-on skills in Python, PySpark, and SQL.
- Experience with Apache Spark, Delta Lake, and Databricks Jobs or Workflows.
- Solid understanding of Lakehouse and medallion architectures.
- Experience building and supporting production-grade ETL/ELT pipelines.
- Experience with cloud data platforms such as Azure, AWS, or GCP.
- Understanding of data quality, monitoring, troubleshooting, and performance optimization.
- Experience with DevOps practices including Git, CI/CD, testing, and deployment processes.
- Strong problem-solving skills and interest in improving and automating existing processes.
- Experience with or understanding of Unity Catalog, Auto Loader, Lakeflow, Structured Streaming, or Databricks Asset Bundles.
- Exposure to cloud-native data and integration services and streaming technologies such as Kafka, Event Hubs, or MQTT.
- Exposure to LLMs, AI agents, MCP, Databricks Genie, RAG, or Model Serving.
- Experience automating data-platform monitoring, troubleshooting, or operational workflows is preferred.
- Relevant Databricks certification is preferred.
- Candidates with less Databricks experience may apply if they have strong practical knowledge of Spark, PySpark, SQL, and modern Lakehouse architectures.
- Ability to collaborate, share ideas and knowledge, propose practical solutions, and experiment meaningfully while maintaining production reliability.
Tech Stack
Categories
Data Engineering