6 months ago
Base Salary
$230k - $265k/yr
Responsibilities
- Design and build scalable data pipelines and lakehouse infrastructure using PySpark, Databricks, Airflow, and AWS.
- Create abstractions, self-service tooling, and documentation to improve the data platform experience for Engineering, Data Science, and Product teams.
- Own core data pipelines and models supporting internal dashboards, ML models, and customer-facing products.
- Administer Databricks workspaces, including access management, performance monitoring, configuration optimization, and pipeline availability.
- Lead initiatives involving data quality, testing, observability, cost efficiency, ingestion scaling, data lineage, and migrations such as Databricks SQL pipelines to dbt.
- Serve as the primary engineering partner for Data Science and collaborate with backend engineers on performant data models and product development.
- Share best practices and mentor engineers working on data-centric systems.
Requirements
- At least 4 years of software engineering experience with a strong background in data infrastructure, pipelines, and distributed systems.
- Advanced proficiency in Python and SQL.
- Hands-on Spark development experience.
- Expertise with AWS, including S3 and RDS, Databricks, Airflow, and lakehouse architectures.
- Hands-on experience with Hadoop, Hive, Kafka or similar streaming platforms, Delta Lake or Iceberg, and Trino or Presto.
- Familiarity with ingestion frameworks, developer-experience tooling, data versioning, lineage, partitioning, and clustering.
- Strong problem-solving skills, ownership mindset, and communication and collaboration abilities.
- Experience with AWS infrastructure using Terraform, observability tools such as Datadog, cloud cost tracking, financial systems, ML infrastructure, feature stores such as Tecton, or internal and open-source data tooling is beneficial.
Benefits
- Medical, dental, and vision insurance
- Work-from-home flexibility
- Unlimited PTO
- Commuter benefits
- Free lunches
- Paid parental leave
- 401(k)
- Employee assistance program
- Equity grant
Tech Stack
Apache AirflowApache HadoopApache HiveApache KafkaApache SparkAWSDatabricksDatadogdbtPrestoPythonSQLTerraform
Categories
Data EngineeringDevOps
About Parafin
Parafin’s infrastructure powers financial services for marketplaces, vertical SaaS, payment platforms, and more. With a single integration, platforms can launch a full suite of financial products for their small business sellers, including capital, spend management, and savings tools. These services provide access to financing and help businesses thrive by leveraging real-time performance data to offer customized financial solutions.