Base Salary
$225k - $300k/yr
Responsibilities
- Lead development of DataHub’s Platform framework, which connects diverse data systems and powers metadata collection
- Build scalable and fault-tolerant ingestion systems for enterprise-scale metadata
- Design clean APIs for the connector ecosystem
- Build event-driven architectures for real-time metadata processing
- Develop schema mapping between diverse systems and DataHub’s unified model
- Build versioning systems for AI assets including training data, model weights, and embeddings
- Design and scale distributed systems, indexing, storage, and data architectures for online services
- Set up monitoring and alerting for production services
Requirements
- 8+ years of experience building production-grade distributed systems
- Advanced Python and API design expertise
- Experience with high-scale data processing or integration frameworks
- Strong systems knowledge and distributed architecture experience
- Proven ability to solve complex technical challenges
- Experience building and maintaining online applications serving live traffic at 100+ QPS
- Experience setting up monitoring and alerting for services
- Experience designing indexing, storage, and data architectures for large-scale online services
- Experience designing and scaling distributed systems
- Hands-on experience developing with LLMs and applying scalable LLM development best practices
- Very strong experience with at least one of Java, Scala, Kotlin, C#, or Go is strongly preferred
- Experience with Python, TypeScript, or Node.js is a plus
- Experience with AWS, Kubernetes, and Docker
- Experience with DataHub or similar metadata/ETL frameworks such as Airflow, Airbyte, or dbt is a bonus
- Open-source contributions are a bonus
- Experience building services that call LLMs to serve live traffic is a bonus
- Experience fine-tuning LLM-powered applications exposed to end users is a bonus
- Early-stage startup experience is a bonus
Benefits
- Hybrid work arrangement in the Bay Area with 3 days per week in the Palo Alto office
- Equity for every team member
- Monthly coworking stipend and location flexibility
- Medical, dental, and vision coverage with 99% of employee premiums and 65% of dependent premiums covered
- Flexible Spending Accounts and Dependent Care FSAs
- Carrot Fertility benefits and family-forming support for all U.S. employees
- Unlimited PTO and sick leave
Tech Stack
Categories
About DataHub
DataHub transforms enterprise data into trusted context, enabling intelligent decision making by humans and AI agents. As the leading context management platform built on a thriving open-source community of 15,000+ members and adopted by thousands of organizations worldwide, DataHub Cloud delivers AI-powered discovery, governance, and observability in a unified platform, ensuring that context is always relevant, reliable, and continuously refreshed across the entire data estate. DataHub is backed by Bessemer Venture Partners, LinkedIn, and 8VC. Our key differentiators: * Scalability: DataHub offers best-in-class enterprise-grade scalability in connecting to over 100 data sources, offering an embeddable connector framework, and ingesting large volumes and high velocity of metadata. * Extensibility: DataHub’s highly extensible metadata model offers easy flexibility in adapting to an organization’s unique data landscape, entities, relationships, ownership, and custom metadata descriptors. * Completeness: DataHub Cloud’s unified platform uses AI-based enhancements and automations for discovery & understanding, quality management, and collaborative governance, allowing users and AI Agents to confidently use and manage data and AI assets. * Open-Source Advantage: Customers of DataHub benefit from the joint innovation, peer support, and growing skill base of an energized community of over 15,000 DataHub practitioners. The managed service, DataHub Cloud, offers dedicated support, improved performance and availability, and secure deployment options to ease adoption across an enterprise. For engineering teams deploying AI in production, DataHub delivers unified context infrastructure across all AI & data assets with enterprise-grade performance.