Data Platform Engineer
Impact.com4 hours ago
Cape Town, South AfricaMid Level / Senior
Responsibilities
- Own data platform components and a production pipeline domain end to end, including design, implementation, reliability, performance, and cost.
- Build scalable batch, ETL/ELT, and streaming pipelines using Spark, Dataproc, Kafka, Pub/Sub, BigQuery, and other data stores.
- Configure and maintain platform infrastructure including Dataproc, Kafka, Airflow/Astronomer, BigQuery, and Cloud Storage.
- Tune query, partitioning, resource, and streaming performance for throughput, reliability, and cost.
- Define and track SLOs for freshness, success, and latency, and implement monitoring, alerting, and data quality checks.
- Participate in Tier 2 on-call response and independently troubleshoot and resolve production incidents.
- Implement access control, secrets management, encryption, audit logging, lineage, retention, governance, and self-service capabilities.
- Write tested, maintainable code and maintain CI/CD workflows, runbooks, architecture documentation, platform standards, and data contracts.
- Mentor Associate Data Platform Engineers through pairing and code review, and communicate technical trade-offs clearly.
Requirements
- 3–5 years of experience in data platform, data engineering, or backend/distributed systems engineering, including ownership of production systems at TB scale or systems with stringent SLAs, multiple downstream consumers, or compliance requirements.
- Strong Python and production experience with a JVM language, with the ability to write production-quality code with error handling, logging, and testing; Scala is the primary pipeline language.
- Hands-on depth in distributed batch processing with Spark or streaming with Kafka or Pub/Sub, plus working familiarity with the other area.
- Advanced SQL skills, including complex queries, window functions, CTEs, and query optimization.
- Experience with workflow orchestration such as Airflow, including scheduling, dependencies, retry logic, and workflow coordination.
- Working fluency with a major cloud platform, ideally GCP, including BigQuery, Dataproc, Cloud Storage, and Pub/Sub.
- Experience with Git, CI/CD, automated testing, code review, monitoring, alerting, incident response, and on-call operations.
- Experience with data-system security and governance practices including access control, secrets management, encryption, and least-privilege design.
- Clear written and verbal communication and experience or aptitude for mentoring less experienced engineers.
- Infrastructure-as-code experience with Terraform or an equivalent tool is advantageous.
- Experience migrating legacy big-data platforms to cloud-native alternatives is advantageous.
- Familiarity with dbt, Looker, semantic-layer concepts, and dbt orchestration through Astronomer Cosmos is advantageous.
- Functional Scala experience with cats-effect and the Typelevel ecosystem, or JVM tuning experience, is advantageous.
- Experience with Grafana, Datadog, Cloud Monitoring, Dataflow, Apache Beam, SingleStore, or BigTable is advantageous.
- A bachelor's degree in Computer Science, Engineering, Mathematics, or a related field and digital marketing technology experience are advantageous.
Benefits
- Hybrid work in Cape Town with two days per week in the office.
- Responsible PTO policy and flexible working environment.
- Up to 12 fully covered therapy or coaching sessions per year, with additional dependent coverage.
- Monthly gym reimbursement.
- Restricted Stock Units with a three-year vesting schedule, pending Board approval.
- Free Coursera subscription and PXA courses.
- 26 weeks of fully paid parental leave for the primary caregiver and 13 weeks for the secondary caregiver.
- Technology stipend for a home office and monthly internet allowance.
Tech Stack
Apache AirflowApache BeamApache KafkaApache SparkDatadogdbtGitGoogle BigQueryGrafanaPythonScalaSingleStoreSQLTerraform
Categories
Data EngineeringDevOps
About Impact.com
Impact.com builds a SaaS platform for partnership and affiliate marketing, used by brands to manage affiliates, influencers, content publishers, and referrals, with tools for tracking, attribution, commissioning, and fraud detection. Its revenue comes from software subscriptions and related services. Founded in 2008 and headquartered in Santa Barbara, California, it is privately held and counts customers such as Walmart, Uber, Shopify, Lenovo, L’Oréal, and Fanatics.