Senior Databricks Data Engineer (Data Migration Project)
ERNI Philippines4 months ago
Mandaluyong City, PhilippinesSenior
Responsibilities
- Design, develop, optimize, and support automated data pipelines, ETL/ELT workflows, data models, curated data layers, and data products using Azure Databricks, Delta Lake, PySpark, and SQL.
- Build scalable ingestion, transformation, validation, reconciliation, exception-handling, and publishing processes for structured, semi-structured, and unstructured data.
- Clarify requirements with business stakeholders, product owners, data consumers, analysts, and subject matter experts, then translate them into technical specifications, mappings, models, and implementation plans.
- Design and implement data marts, analytical datasets, customized data extracts, and consumption-ready data layers.
- Develop reusable data engineering frameworks, patterns, and components that improve delivery speed, reliability, monitoring, and maintainability.
- Implement data quality, lineage, auditability, security, privacy, access management, compliance, logging, and monitoring controls.
- Monitor pipeline health, performance, cost, and reliability; troubleshoot production incidents and perform root-cause analysis.
- Optimize Databricks jobs, Spark workloads, SQL queries, Delta tables, partitioning strategies, and orchestration patterns.
- Contribute to lakehouse architecture, ingestion patterns, transformation standards, orchestration, metadata management, and operational support.
- Support batch and scheduled processing, with near-real-time and frequent-refresh patterns as an advantage.
Requirements
- Minimum 8 years of professional data engineering experience, including enterprise-scale data pipelines, data platforms, ETL/ELT processes, and analytics-ready datasets.
- Senior-level hands-on experience with Azure and Azure Databricks.
- Strong experience with Databricks, PySpark, Spark SQL, Delta Lake, and SQL.
- Experience with cloud-native ingestion, transformation, orchestration, monitoring, performance tuning, production support, data modeling, data marts, and curated data layers.
- Experience with source-to-target mapping, data profiling, data validation, reconciliation, and data quality management.
- Experience with orchestration and scheduling tools such as Azure Data Factory, Databricks Workflows, Airflow, or similar technologies.
- Understanding of data governance, access control, metadata, lineage, security, privacy, and compliance in cloud data environments.
- Strong troubleshooting, problem-solving, root-cause analysis, communication, stakeholder collaboration, and independent-working skills.
- Preferred experience with Spark Structured Streaming, Auto Loader, Kafka, Event Hubs, change data capture, incremental ingestion, streaming tables, and near-real-time refresh patterns.
- Preferred experience with lakehouse data layers, CI/CD, DevOps, automated testing, version control, deployment pipelines, environment management, and Azure data services.
- Experience in regulated or data-sensitive enterprise environments and support for business intelligence, analytics, machine learning, or operational reporting use cases is preferred.
Benefits
- Hybrid work setup with a dayshift schedule of 9 AM to 6 PM.
- Office location is Mandaluyong or BGC at the client's office.
- Private HMO and insurance coverage from day one.
- Government-mandated benefits including 13th-month pay.
- Free technical and non-technical training and certification opportunities.
- Career stability, project opportunities, professional development, recognition, and promotion opportunities.
- Five-year milestone benefit including a complimentary trip to any European ERNI site.
- Free snacks and coffee, baby basket, fruit basket, and company social, family, summer, leisure, and year-end events.
Tech Stack
Categories
Data Engineering
About ERNI Philippines
ERNI Philippines provides software engineering and IT consulting services, delivering custom development, DevOps, UX/UI, and agile transformation for clients in sectors such as MedTech, finance, industry, and transportation. It operates as part of the Swiss-founded ERNI Group (established in 1994; headquarters in Lucerne), running a service center in Metro Manila that supports projects across Europe, APAC—including the local Philippine market—and the USA.