TaskUs

Senior Data Engineer

TaskUs
Apply
12 hours ago
Chennai, IndiaSenior

Responsibilities

  • Build and maintain high-throughput ETL/ELT pipelines ingesting data from multiple sources into a lakehouse.
  • Develop modular, reusable data transformations using dbt and PySpark with strict CI/CD practices.
  • Implement data SLAs, monitoring, alerting, and reliability systems for data drift and pipeline failures.
  • Prepare feature-ready datasets in collaboration with BI teams and data scientists.
  • Tune SQL and Spark query plans to reduce job cost and latency.
  • Enable local development and testing of SQL transformations using DuckDB.
  • Build cost-efficient micro-pipelines for client reporting and data-quality checks using DuckDB and lightweight containers such as Lambda or ECS tasks.
  • Explore DuckDB for embedded analytics and edge processing in local site offices.
  • Choose between distributed and lightweight compute patterns to optimize cost and latency.
  • Use DuckDB to query Parquet or Iceberg data stored in S3 or Azure Blob.
  • Manage data infrastructure and Kubernetes resources, including Pods, Deployments, Services, ConfigMaps, and Secrets.

Requirements

  • At least 5 years of data engineering experience.
  • Expert-level proficiency in Python, SQL, and PySpark.
  • Hands-on experience with Databricks and Delta Lake or Snowflake, including lakehouse and Medallion Architecture concepts.
  • Advanced experience with dbt, including version control, testing, and documentation of data models.
  • Experience building complex, idempotent DAGs with Apache Airflow or Prefect.
  • Experience with DuckDB for local development, unit testing, embedded querying, or datasets up to 100GB.
  • Experience with Terraform or CloudFormation for infrastructure as code.
  • Understanding of Kubernetes and its Pods, Deployments, Services, ConfigMaps, and Secrets.
  • Ability to select appropriate heavyweight or lightweight compute for data workloads.
  • Streaming experience with Kafka, Kinesis, or Spark Streaming is preferred.
  • Experience querying Parquet or Iceberg files in S3 or Azure Blob using DuckDB is preferred.

Benefits

  • Remote work arrangement.
  • Competitive industry salaries and comprehensive benefits packages.
  • Employee well-being support through wellness and Total Rewards programs.
  • Inclusive workplace with reasonable accommodations and equal access to opportunities.
  • Professional growth and internal mobility opportunities.

Tech Stack

Apache AirflowApache KafkaDatabricksdbtKubernetesPythonSnowflakeSQLTerraform

Categories

Data Engineering
TaskUs

About TaskUs

10,000+ employees

TaskUs provides outsourced digital operations and customer experience services for technology and internet companies, including customer support, trust and safety/content moderation, and AI data services. It operates a global delivery network and earns revenue through multi‑year, volume‑based BPO contracts. Founded in 2008 and headquartered in New Braunfels, Texas, TaskUs is a public company listed on Nasdaq.

Contact me