1 day ago
Base Salary
$89k - $166k/yr
Responsibilities
- Design, develop, and maintain scalable batch and real-time data pipelines using PySpark, SQL, Delta Lake, and Auto Loader.
- Build ETL/ELT frameworks for ingesting, transforming, and processing large-scale datasets.
- Implement data quality checks, monitoring, reliability improvements, and pipeline performance optimization.
- Design and manage Databricks Lakehouse solutions using Medallion Architecture.
- Develop and optimize Delta Lake tables and perform Databricks performance tuning.
- Configure Unity Catalog and implement catalog, schema, table, row-level, and column-level security controls.
- Enable data lineage, auditing, and compliance across the data platform.
- Develop and maintain Databricks Workflows and manage deployments with Databricks Asset Bundles.
- Implement CI/CD pipelines and provision infrastructure using Terraform.
- Prepare datasets for machine learning and generative AI, support feature engineering and feature stores, and enable vector search and retrieval.
- Collaborate with data architects, data scientists, AI teams, business stakeholders, and application teams to operationalize ML workloads.
Requirements
- 5–8 years of experience in data engineering.
- Strong proficiency in Python, PySpark, and SQL.
- Hands-on experience with Databricks Lakehouse Platform, Delta Lake, Unity Catalog, Databricks Workflows, Structured Streaming, and Auto Loader.
- Experience designing and supporting large-scale enterprise data platforms.
- Understanding of data modeling, data warehousing, and ETL/ELT design patterns.
- Experience implementing CI/CD pipelines, version control, and automated testing practices.
- Hands-on experience with Git, GitHub or GitLab, Terraform, and pytest.
- Experience with at least one major cloud platform: Azure, GCP, or AWS.
Benefits
- Comprehensive benefits package for regular full-time employees, including paid time off, company holidays, personal days, and sick leave.
- Medical, dental, and vision coverage, or provincial healthcare coordination in Canada.
- Retirement savings plans such as 401(k) in the U.S. or RRSP in Canada.
- Life and disability insurance and employee assistance programs.
- Onsite work at the customer location in Troy, MI at least four days per week.
Categories
Data Engineering
About Capgemini
Capgemini is a global IT services and consulting firm that delivers strategy, cloud, AI, software engineering, and managed services to large enterprises and public-sector clients. Founded in 1967 and headquartered in Paris, it is publicly traded on Euronext Paris and operates in 50+ countries. The group expanded its engineering capabilities by acquiring Altran in 2020, now operating as Capgemini Engineering.