Databricks Data Engineer
Hitachi Solutions29 days ago
Pune, IndiaSenior
Responsibilities
- Architect and implement complex data systems with minimal oversight.
- Design, develop, enhance, and maintain complex Databricks data pipeline products.
- Build data products supporting business-critical operations and large-scale analytics applications.
- Partner with analytics, data science, and engineering teams to solve data needs and challenges.
- Identify opportunities to improve and expand platform capabilities with team leadership.
- Stay current with developments in data engineering technology and contribute to the organization’s technical maturity.
Requirements
- 5+ years of data engineering experience, including 4+ years designing and building Databricks data pipelines.
- 4+ years of hands-on experience with Python, PySpark, or SparkSQL, plus 4+ years of Spark experience.
- 4+ years of experience with big data pipelines or DAG tools such as Airflow, dbt, Data Factory, or similar.
- 4+ years implementing big data solutions in a cloud ecosystem, including data or Delta Lakes.
- 5+ years of relevant software development experience with Python, Django or Flask, FastAPI or another standard API framework, and relational databases using SQL or ORM.
- Experience with Azure is preferred; AWS, GCP, or other cloud platforms are also considered.
- Scala experience is a plus.
- Preferred experience includes Microsoft Fabric, Fabric Pipelines, database design, advanced SQL queries, Kimball/star schema modeling, medallion architecture, Kafka, EventHub, database deployment pipelines, unit testing, or data quality frameworks.
Benefits
- Full-time role with competitive compensation and benefits.
- Performance bonuses and regular performance reviews.
- Regular training and opportunities for personal and professional growth.
- Challenging and diverse work across multiple industries in a collaborative environment.
Tech Stack
Apache AirflowApache KafkaApache SparkAWSAzureDatabricksdbtDjangoFastAPIFlaskGoogle Cloud PlatformPythonScalaSQL
Categories
Data Engineering
About Hitachi Solutions
Hitachi Solutions provides consulting, implementation, and managed services around Microsoft Dynamics 365, Azure, Power Platform, data, and AI for enterprises and public-sector organizations. It helps customers modernize finance and operations, build low-code apps, and run cloud workloads, combining advisory services with application support and change management. The company is part of Hitachi, Ltd., giving it access to a broad global ecosystem while operating regional teams, including in Portugal.