2 months ago
Remote, HungarySenior
Responsibilities
- Design and implement scalable batch and streaming data pipelines using Apache Spark, Kafka, and Flink.
- Build and maintain Bronze/Silver/Gold medallion architecture using Delta Lake or Apache Iceberg.
- Develop and optimize SQL and PySpark transformations for large-scale datasets.
- Integrate structured, semi-structured, and unstructured data sources into the lakehouse.
- Collaborate with data architects on physical and logical data models.
- Implement data quality checks and monitoring using Great Expectations or dbt tests.
- Write infrastructure-as-code for pipeline environments using Terraform and Helm.
- Participate in code reviews and enforce engineering standards and best practices.
- Troubleshoot pipeline failures, performance bottlenecks, and data incidents.
- Mentor junior and mid-level data engineers and contribute to internal knowledge sharing.
Requirements
- At least 6 years of data engineering experience with enterprise-scale delivery.
- Expert proficiency in Python and SQL, with required PySpark experience.
- Hands-on experience with Apache Spark, Delta Lake, or Apache Iceberg.
- Experience with Apache Airflow, Prefect, or Dagster.
- Strong knowledge of AWS Glue, Azure Data Factory, or GCP Dataflow.
- Proficiency with Git, CI/CD pipelines, and Docker/Kubernetes containerization.
- Experience using dbt for transformation-layer management.
- Bachelor's degree in Computer Science, Engineering, or a related technical field.
- Preferred: experience with Databricks, Snowflake, or Apache Hudi.
- Preferred: knowledge of streaming architectures and Apache Kafka.
- Preferred: Databricks Certified Data Engineer or AWS Data Analytics Specialty certification.
Benefits
- Remote work is indicated by the #LI-Remote posting tag.
- Dynata describes an inclusive and accessible work environment and provides accommodations by request throughout the selection process.
Tech Stack
Apache AirflowApache FlinkApache KafkaApache SparkDatabricksdbtDockerGitHelmKubernetesPythonSnowflakeSQLTerraform
Categories
Data Engineering
About Dynata
Dynata provides first-party survey data and an insights platform used by market researchers, advertisers, consultants, and enterprises to target audiences and measure outcomes. The privately held company, headquartered in Westport, Connecticut, was created from the merger of Research Now and Survey Sampling International and rebranded as Dynata in 2019, and says it serves more than 6,000 customers across North America, South America, Europe, and Asia-Pacific.