Data Engineering Architect, Senior
Bloomberg Industry Group2 months ago
Base Salary
$180k - $220k/yr
Responsibilities
- Partner with product analytics stakeholders to translate KPIs and data requirements into scalable production datasets.
- Design, build, operate, monitor, and troubleshoot end-to-end data pipelines from ingestion through serving.
- Define and evolve Product Analytics Lakehouse architecture for scalability, performance, reliability, governance, maintainability, and cost efficiency.
- Build and maintain well-modeled Gold-layer datasets for analytics, experimentation, and AI/ML use cases.
- Create and drive adoption of reusable data engineering patterns, frameworks, standards, and guardrails.
- Own production data quality and reliability, including validation, monitoring, SLAs, and incident resolution.
- Productionize prototype datasets and logic created by analytics partners.
- Lead Databricks workload migration, modernization, and coexistence strategies.
- Make and communicate architectural tradeoffs across technical and business considerations.
- Provide architectural guidance and hands-on technical leadership across Product Analytics.
- Mentor engineers and other technical contributors and raise engineering standards without formal authority.
Requirements
- Strong experience building and operating data pipelines with SQL and Python in modern cloud environments.
- Deep SQL expertise, including complex transformations, data modeling, query optimization, and performance tuning at scale.
- At least 2 years of recent hands-on production experience with Databricks.
- Strong hands-on experience with Spark/PySpark and distributed production data processing.
- Understanding of Lakehouse architecture, ELT, and layered bronze/silver/gold data models.
- Experience designing dimensional or domain-oriented analytics data models.
- Experience with schema design, migrations, database and data engineering best practices, and performance optimization.
- Ability to own high-impact architecture and engineering decisions from design through production in environments with limited structure.
- Experience establishing reusable data frameworks, standards, and guardrails adopted beyond an individual project or pipeline.
- Experience owning production data systems, including monitoring, debugging, and resolving pipeline failures.
- Experience translating ambiguous stakeholder or analyst requirements into scalable data solutions.
- Strong technical judgment regarding scalability, reliability, cost, maintainability, governance, and developer experience.
- Experience evolving or migrating legacy data platforms while maintaining downstream continuity.
- Strong understanding of metadata, lineage, access controls, data quality, and data governance.
- Strong project execution experience and familiarity with agile development practices.
- Excellent written and verbal communication skills and the ability to influence technical direction across teams.
- Experience developing source-controlled pipelines in a CI/CD environment.
- Product analytics experience involving event tracking, user behavior, or experimentation is preferred.
- Bachelor’s degree in computer science or a related discipline, or equivalent experience; a master’s degree is preferred.
- 10+ years of experience in data engineering, data architecture, software engineering, or related fields.
- Experience building and maintaining scalable cloud-based data pipelines and datasets with recent Databricks and cloud infrastructure experience.
- Ability to design and evolve data architectures, data models, and production pipelines supporting analytics at scale.
- Experience taking data solutions from concept through production, monitoring, debugging, and ongoing maintenance.
- Ability to operate independently at an architectural level, make high-impact decisions, influence engineering practices, and raise data platform standards.
Tech Stack
Categories
Data Engineering
About Bloomberg Industry Group
Bloomberg Industry Group empowers professionals in government, law, tax, and accounting with the news, data, and analysis they need to take decisive action and make the most of every opportunity. Bloomberg Industry Group is an affiliate of Bloomberg L.P. For more information, visit bloombergindustry.com.