Senior Staff Engineer, Data Platform
Qualified Health PBC5 hours ago
Base Salary
$215k - $235k/yr
Responsibilities
- Architect and evolve the Databricks lakehouse, including medallion layering, Delta Lake, Unity Catalog governance, and workspace and compute strategy.
- Design, build, optimize, and operate Spark and SQL pipelines on Databricks, including incremental processing, data quality, performance tuning, and cost management.
- Define data contracts, quality checks, access controls, and PHI-aware governance in partnership with security and compliance.
- Build governed data foundations for AI and LLM products, including LLM-ready datasets, retrieval and vector-store pipelines, evaluation data, and data interfaces.
- Design durable Temporal workflows for AI pipelines, including scheduled and watermarked pulls, file ingestion, long-running extraction and enrichment jobs, retries, idempotency, versioning, and backfills.
- Define team practices for AI coding agents, including tooling, prompts, guardrails, code review, and testing strategies.
- Set platform-wide technical standards, write and review designs, and make long-term build-versus-buy decisions.
- Mentor engineers across teams, unblock other pods, and communicate technical trade-offs to technical and non-technical stakeholders.
Requirements
- Bachelor’s degree in Computer Science, Electrical Engineering, Mathematics, or a related field.
- At least 12+ years of software or data engineering experience, including designing and operating production data platforms at scale and setting architecture across multiple teams.
- Required experience working in a highly regulated domain such as healthcare, insurance, or finance.
- Deep hands-on Databricks expertise is highly preferred; Snowflake or Microsoft Fabric experience is an aligned alternative.
- Experience with Spark, Delta Lake, Unity Catalog, Workflows or equivalent, and large-scale performance and cost tuning.
- Experience building pipelines for ML or LLM applications, such as LLM extraction, embeddings and vector search, feature pipelines, or evaluation datasets.
- Fluency in Python and SQL, with sound software engineering practices including testing, CI/CD, infrastructure as code, and code review.
- Demonstrated practical use of agentic coding tools such as Claude Code in real delivery work.
- Experience owning ambiguous cross-team problems end to end and influencing without direct authority.
- Clear written and verbal communication, including authoring technical design documents.
- Healthcare data experience involving claims, EHR, FHIR, HL7, or HIPAA-compliant PHI handling is desirable.
- Databricks certifications or experience with Delta Live Tables, Lakeflow, or Databricks Asset Bundles is desirable.
- Experience with Airflow, Dagster, Prefect, cloud infrastructure, Terraform, and Kubernetes is desirable.
- Contributions to open-source data, workflow, or AI infrastructure projects are desirable.
Benefits
- Competitive salary of $215,000–$235,000 plus equity eligibility.
- Medical, dental, and vision insurance.
- Flexible working hours.
- Hybrid work options.
- Inclusive, mission-driven environment with opportunities to work on healthcare AI infrastructure.
Tech Stack
Apache AirflowApache SparkAWSAzureDatabricksGoogle Cloud PlatformKubernetesPythonSnowflakeSQLTerraform
Categories
Data Engineering
About Qualified Health PBC
Qualified Health PBC builds an enterprise AI platform for health systems, combining workflow automation, generative AI agent development, clinical safeguards, real-time monitoring, and end-to-end governance. The company sells software and partnership-based deployment services to drive measurable clinical and financial outcomes for hospitals and health networks. Headquartered in Palo Alto, it reports reach to over 500,000 users across top U.S. health systems and is backed by investors including NEA, Transformation Capital, and SignalFire.