6 months ago
Palo Alto, CA, USAStaff+
Base Salary
$225k - $300k/yr
Responsibilities
- Lead development of DataHub’s Platform framework, which connects diverse data systems and powers metadata collection
- Build scalable and fault-tolerant ingestion systems for enterprise-scale metadata
- Design clean APIs for the connector ecosystem
- Build event-driven architectures for real-time metadata processing
- Develop schema mapping between diverse systems and DataHub’s unified model
- Build versioning systems for AI assets including training data, model weights, and embeddings
- Design and scale distributed systems, indexing, storage, and data architectures for online services
- Set up monitoring and alerting for production services
Requirements
- 8+ years of experience building production-grade distributed systems
- Advanced Python and API design expertise
- Experience with high-scale data processing or integration frameworks
- Strong systems knowledge and distributed architecture experience
- Proven ability to solve complex technical challenges
- Experience building and maintaining online applications serving live traffic at 100+ QPS
- Experience setting up monitoring and alerting for services
- Experience designing indexing, storage, and data architectures for large-scale online services
- Experience designing and scaling distributed systems
- Hands-on experience developing with LLMs and applying scalable LLM development best practices
- Very strong experience with at least one of Java, Scala, Kotlin, C#, or Go is strongly preferred
- Experience with Python, TypeScript, or Node.js is a plus
- Experience with AWS, Kubernetes, and Docker
- Experience with DataHub or similar metadata/ETL frameworks such as Airflow, Airbyte, or dbt is a bonus
- Open-source contributions are a bonus
- Experience building services that call LLMs to serve live traffic is a bonus
- Experience fine-tuning LLM-powered applications exposed to end users is a bonus
- Early-stage startup experience is a bonus
Benefits
- Hybrid work arrangement in the Bay Area with 3 days per week in the Palo Alto office
- Equity for every team member
- Monthly coworking stipend and location flexibility
- Medical, dental, and vision coverage with 99% of employee premiums and 65% of dependent premiums covered
- Flexible Spending Accounts and Dependent Care FSAs
- Carrot Fertility benefits and family-forming support for all U.S. employees
- Unlimited PTO and sick leave
Tech Stack
Categories
BackendData Engineering
About DataHub
DataHub builds an open-core metadata platform and DataHub Cloud, a managed SaaS for data cataloging, lineage, governance, and observability used by data and AI teams. The company commercializes the open-source DataHub project; customers include Apple, Netflix, Visa, and CVS Health. Founded in 2021 and headquartered in Palo Alto, it is privately held and backed by Bessemer Venture Partners, LinkedIn, and 8VC.