12 hours ago
Chennai, IndiaSenior
Responsibilities
- Build and maintain high-throughput ETL/ELT pipelines ingesting data from multiple sources into a lakehouse.
- Develop modular, reusable data transformations using dbt and PySpark with strict CI/CD practices.
- Implement data SLAs, monitoring, alerting, and reliability systems for data drift and pipeline failures.
- Prepare feature-ready datasets in collaboration with BI teams and data scientists.
- Tune SQL and Spark query plans to reduce job cost and latency.
- Enable local development and testing of SQL transformations using DuckDB.
- Build cost-efficient micro-pipelines for client reporting and data-quality checks using DuckDB and lightweight containers such as Lambda or ECS tasks.
- Explore DuckDB for embedded analytics and edge processing in local site offices.
- Choose between distributed and lightweight compute patterns to optimize cost and latency.
- Use DuckDB to query Parquet or Iceberg data stored in S3 or Azure Blob.
- Manage data infrastructure and Kubernetes resources, including Pods, Deployments, Services, ConfigMaps, and Secrets.
Requirements
- At least 5 years of data engineering experience.
- Expert-level proficiency in Python, SQL, and PySpark.
- Hands-on experience with Databricks and Delta Lake or Snowflake, including lakehouse and Medallion Architecture concepts.
- Advanced experience with dbt, including version control, testing, and documentation of data models.
- Experience building complex, idempotent DAGs with Apache Airflow or Prefect.
- Experience with DuckDB for local development, unit testing, embedded querying, or datasets up to 100GB.
- Experience with Terraform or CloudFormation for infrastructure as code.
- Understanding of Kubernetes and its Pods, Deployments, Services, ConfigMaps, and Secrets.
- Ability to select appropriate heavyweight or lightweight compute for data workloads.
- Streaming experience with Kafka, Kinesis, or Spark Streaming is preferred.
- Experience querying Parquet or Iceberg files in S3 or Azure Blob using DuckDB is preferred.
Benefits
- Remote work arrangement.
- Competitive industry salaries and comprehensive benefits packages.
- Employee well-being support through wellness and Total Rewards programs.
- Inclusive workplace with reasonable accommodations and equal access to opportunities.
- Professional growth and internal mobility opportunities.
Tech Stack
Categories
Data Engineering
About TaskUs
TaskUs provides outsourced digital operations and customer experience services for technology and internet companies, including customer support, trust and safety/content moderation, and AI data services. It operates a global delivery network and earns revenue through multi‑year, volume‑based BPO contracts. Founded in 2008 and headquartered in New Braunfels, Texas, TaskUs is a public company listed on Nasdaq.
