17 days ago
Bengaluru, IndiaStaff+
Responsibilities
- Lead the technical design and delivery of batch and near-daily data pipelines from raw inputs through curated outputs and publication to the Master Data Layer.
- Profile telemetry and diverse source data, identify gaps, and translate source formats and business rules into reliable processing logic.
- Design and operate real-time and streaming workflows with event validation, schema evolution, ordering, deduplication, replay handling, latency management, backpressure, and failure recovery.
- Build parsers, configuration-driven workflows, session and event logic, business rules, schema mappings, data-stitching logic, and transformation pipelines across mobile, CTV, desktop, and web products.
- Deliver processing for streaming/video, e-commerce, advertising, application usage, web usage, purchase activity, and incremental dataset updates.
- Integrate content metadata, reference data, classification inputs, and model-based enrichment in partnership with machine-learning and platform teams.
- Build reusable onboarding frameworks supporting new applications, platforms, data sources, markets, historical backfills, reprocessing, and safe daily refreshes.
- Orchestrate and operate workflows using Airflow and AWS workflow services, monitoring schedules, dependencies, retries, failures, and data availability.
- Implement data-quality checks, reconciliation, freshness and volume monitoring, schema and completeness validation, duplicate and range checks, stitching validation, and cross-layer consistency controls.
- Create dashboards, alerts, and job-level telemetry for pipeline health, data quality, performance, processing costs, and operational reliability.
- Optimize distributed processing and cloud data layouts for correctness, scalability, runtime, and cost.
- Contribute to design and code reviews, testing strategy, release readiness, runbooks, technical documentation, data contracts, schemas, and operational ownership boundaries.
- Provide technical direction and mentoring, communicate risks and dependencies, and lead multiple complex cross-functional projects from definition through production delivery.
- Apply AI-assisted engineering and operations responsibly for data exploration, code and test generation, classification, anomaly detection, and workflow automation.
- Handle sensitive data responsibly and follow Nielsen security, privacy, access-control, and engineering practices.
Requirements
- Typically 10–15 years of relevant data engineering or data-processing experience, or equivalent depth of expertise delivering complex production data systems.
- Bachelor’s degree in Computer Science, Engineering, Information Systems, or a related field, or equivalent practical experience.
- Expert-level SQL and strong programming skills in Python and/or Scala/Java.
- Substantial hands-on experience building distributed data-processing workloads with Apache Spark or a comparable framework.
- Proven experience designing, deploying, and supporting production data pipelines and data-lake or lakehouse workflows on AWS or a comparable cloud platform.
- Practical experience with object storage, columnar formats, workflow orchestration, metadata and catalog services, and query engines.
- Strong understanding of data semantics and modeling, source-to-target lineage, schema evolution, data contracts, event-driven and batch-processing patterns, partitioning, incremental loads, backfills, and idempotent or recoverable processing.
- Hands-on knowledge of real-time event-processing concepts including delivery guarantees, event-time handling, ordering, deduplication, replay, latency, and backpressure.
- Ability to apply AI and ML appropriately to data understanding, processing, quality, and operational efficiency while verifying generated outputs.
- Demonstrated ability to investigate unfamiliar source data, convert business rules into testable processing logic, and diagnose issues across pipeline stages.
- Track record of independently driving multiple complex, cross-functional projects from problem definition through production delivery without formal people-management authority.
- Experience building data-quality checks, reconciliation, monitoring, and alerting for production data products.
- Strong analytical, problem-solving, communication, and technical-judgment skills.
- Preferred experience with telemetry stitching, user activity and digital interaction signals, session or content metrics, content metadata enrichment, entity matching, e-commerce classification, advertising datasets, or ML-assisted data processing.
- Preferred experience with AWS Kinesis, Firehose, Lambda, Step Functions, Glue, Athena, EMR, Grafana, or Datadog.
- Relevant advanced degree or specialized certification is a plus.
Tech Stack
Categories
Data Engineering
About Nielsen
Nielsen provides audience measurement and analytics that quantify who watches, listens, and buys across TV, digital, radio, and streaming. It sells ratings, cross-platform datasets, and advertising effectiveness tools to media owners, advertisers, and agencies on subscription and consulting agreements. Founded in 1923 and headquartered in New York, it operates in 55+ countries and is known for U.S. TV ratings.