Sr. Databricks Data Engineer, Vice President
State Street7 days ago
Base Salary
$120k - $203k/yr
Responsibilities
- Design, build, and test end-to-end data pipelines for streaming, event, and batch ingestion using Azure and Cosmos DB.
- Develop and implement ETL processes for loading transformed data into data lakes.
- Work extensively with Databricks and data warehousing concepts, including Databricks SQL and Delta Lake.
- Design and develop high-throughput, configurable frameworks and libraries.
- Contribute to all stages of the software development lifecycle, including design, coding, integration testing, deployment, and documentation.
- Monitor and troubleshoot data pipelines and DevOps pipelines, including ADO and Harness.
- Implement data-quality checks and validations and perform data analysis and exploration.
- Collaborate with an agile project team and drive change through collaboration, influence, and proof-of-concept demonstrations.
Requirements
- Bachelor's degree in a computer- or IT-related subject.
- 10+ years of overall Big Data pipeline experience.
- 5+ years of hands-on Databricks experience.
- 5+ years of cloud-based development experience, including Azure services, Azure DevOps, Kubernetes, and Docker.
- Hands-on development experience with Databricks SQL and Scala.
- Proficiency with Databricks, Delta Lake, Apache Spark, clusters, jobs, and other Databricks resources.
- Experience with microservices architecture, cloud computing, data warehousing, semi-structured data, and troubleshooting DevOps pipelines.
- Experience using version control such as Git in a multi-developer environment.
- Experience with Avro, JSON, XML, shell scripting, UNIX, TOAD, and SQL Developer.
- Experience with Snowflake is preferred.
- Experience in the financial industry and with agile development methodologies is preferred.
- Strong communication, critical-thinking, problem-solving, collaboration, multitasking, initiative, and self-motivation skills.
Benefits
- Onsite work four days per week and remote work one day per week, during standard business hours aligned to project and stakeholder needs.
- Retirement savings plan with 401(k) company match.
- Medical, dental, vision, life, long-term disability, and other optional insurance coverage.
- Paid vacation, sick leave, short-term disability, and family-care-responsibility leave.
- Employee Assistance Program.
- Eligibility for incentive compensation and certain tax-advantaged savings plans.
- Inclusive development opportunities, flexible work-life support, paid volunteer days, and employee networks.
Tech Stack
Categories
Data Engineering
About State Street
State Street is a global custodian bank and asset manager serving institutional investors with investment servicing, fund accounting, trading/FX, data, and index/active strategies. Its products include State Street Global Advisors’ funds and ETFs, the Charles River Investment Management Solution, and the front-to-back State Street Alpha platform. Founded in 1792 and headquartered in Boston, the company is publicly traded on the NYSE (ticker: STT) and operates across major markets worldwide.