Senior Data Platform Engineer
Sword Health4 months ago
Porto, PortugalSenior
Responsibilities
- Design and evolve Sword’s streaming lakehouse foundation.
- Build and operate distributed streaming pipelines for low-latency, highly reliable data movement.
- Own durable workflows coordinating complex data movement across systems.
- Shape the platform’s API surface for self-service producers and consumers.
- Evaluate and integrate vendor data platforms while assessing architectural trade-offs.
- Develop self-service and agentic interfaces for humans, systems, and AI agents.
- Partner with data engineers and analysts on contracts, governance, and lineage.
- Build and maintain AI-ready data infrastructure for ML and AI-driven products.
- Use AI coding assistants and LLMs to accelerate development, documentation, and code quality.
- Design for audit, compliance, and governance in a regulated healthcare environment.
Requirements
- Experience designing and operating production data platforms at scale, including warehouse, data lake, or lakehouse architectures.
- Hands-on experience with modern lakehouse table formats, preferably Iceberg, with understanding of metadata, snapshots, manifests, compaction, copy-on-write, and merge-on-read.
- Understanding of catalog architectures and trade-offs, including REST, Polaris, Glue, Unity, and Hive catalogs.
- Experience with Snowflake, Starburst, or Databricks at an architectural level.
- Strong experience with distributed processing engines, preferably Flink or alternatively Spark, including job tuning and pipeline debugging.
- Familiarity with durable execution using Temporal, Restate, or similar systems.
- Production experience building and operating REST or gRPC APIs at scale, including contracts, versioning, retries, rate limiting, and observability.
- Strong understanding of Kafka and event-driven architectures, including producers, consumers, partitioning, and delivery semantics.
- Experience working in regulated environments such as healthcare, fintech, or government.
- A platform mindset focused on self-service, API-first systems, and human and agent consumers.
- Preferred experience with open or REST catalogs such as Polaris, Nessie, and Unity.
- Preferred fluency with Prometheus, Grafana, and OpenTelemetry.
- Preferred experience with agentic or AI-facing API surfaces or MCP-style interfaces.
- Preferred experience in HIPAA, FedRAMP, or SOC 2 environments.
- Preferred exposure to dbt, DataHub, or data contract tooling.
- Candidates must demonstrate at least Level 1 AI proficiency under Sword’s AI fluency framework.
Benefits
- Health, dental, and vision insurance.
- Meal allowance, equity shares, remote work allowance, flexible working hours, work-from-home option, discretionary vacation, snacks, and beverages.
- Remote work is supported and the role is based in Portugal.
- The position does not offer relocation assistance; candidates must be based in Portugal and possess a valid EU visa.
Tech Stack
Categories
Data Engineering