Senior Data Platform Engineer
Freshfields Bruckhaus Deringer24 days ago
Raleigh, NC, USASenior / Staff+
Responsibilities
- Manage and maintain the enterprise Data Office platform across Azure, Databricks, Unity Catalog, Collibra, and Immuta.
- Monitor and optimize platform performance, workloads, clusters, SQL warehouses, jobs, storage, compute, capacity, concurrency, autoscaling, and costs.
- Administer workspaces, environments, service connections, secrets, policies, integrations, access controls, entitlements, and Unity Catalog structures.
- Design, build, and maintain secure lakehouse solutions and reusable batch, streaming, ETL, ELT, integration, and ingestion pipelines.
- Implement observability through monitoring, logging, alerting, performance metrics, usage and cost reporting, dashboards, and failure analysis.
- Lead incident investigations, root-cause analysis, remediation, problem management, upgrades, migrations, resilience testing, backup, recovery, and lifecycle management.
- Automate platform deployment and configuration using infrastructure as code, CI/CD, testing, release management, and version control.
- Implement data quality, reconciliation, validation, exception handling, recovery, audit, governance, metadata, lineage, and access controls.
- Maintain technical documentation, runbooks, architecture diagrams, operating procedures, data dictionaries, and knowledge articles.
- Provide technical leadership, coaching, design review, and best-practice guidance to data engineers and platform users.
Requirements
- 5–10 years of hands-on experience in data engineering, data platform engineering, cloud engineering, or a closely related field.
- Proven experience managing production enterprise data platforms, including performance, workloads, storage, compute, access, availability, and cost.
- Experience designing and implementing cloud data lake or lakehouse architectures at scale.
- Experience operating in a Microsoft Azure enterprise environment and collaborating with platform, security, governance, and engineering teams.
- Experience in a regulated environment, preferably legal or financial services, with knowledge of security, privacy, audit, and compliance obligations.
- Advanced hands-on Databricks experience, including Spark, PySpark, Delta Lake, SQL, Workflows, clusters, SQL warehouses, jobs, performance tuning, and cost optimization.
- Hands-on experience administering Unity Catalog, including catalogs, schemas, permissions, lineage, secure data access, and governance.
- Strong experience with Microsoft Azure services, including Azure Data Lake Storage Gen2, Azure Data Factory, Azure Functions, Logic Apps, Key Vault, networking, identity, and monitoring.
- Strong Python and SQL skills for developing, reviewing, troubleshooting, and optimizing production code and queries.
- Experience with integration and ingestion across databases, enterprise applications, files, APIs, streaming, and event-based sources.
- Experience with Collibra or a comparable data catalog and governance platform, and Immuta or a comparable data security platform.
- Knowledge of identity and access management, role-based and attribute-based access control, service principals, managed identities, secrets, and least-privilege design.
- Experience with platform monitoring, observability, capacity management, incident response, root-cause analysis, resilience, backup, and recovery.
- Experience with CI/CD, infrastructure as code, source control, automated testing, and release management using Terraform, Git, Azure DevOps, GitHub Actions, or equivalent tools.
- Desirable experience includes dbt, Apache Airflow, Azure Event Hubs, Spark Structured Streaming, Azure Synapse Analytics, Microsoft Purview, containerization, Kubernetes, enterprise API management, and FinOps practices.
- Strong analytical, troubleshooting, communication, prioritization, ownership, judgment, and knowledge-sharing skills.
Benefits
- The role supports development, test, and production environments and includes operational, engineering, and platform improvement responsibilities.
- The employer provides equal employment opportunities and reasonable accommodation for qualified applicants with disabilities and disabled veterans.
Tech Stack
Categories
Data EngineeringDevOps
About Freshfields Bruckhaus Deringer
Freshfields Bruckhaus Deringer is a global law firm advising corporations, financial institutions, and governments on complex transactions, disputes, and regulatory matters across jurisdictions. Structured as a partnership headquartered in London and founded in 1743, it operates offices across Europe, Asia, the Middle East, and the United States. Its fee-based practice spans M&A, antitrust, arbitration and litigation, finance and capital markets, and compliance and investigations.