Staff Software Engineer (Data Platform)
Arcana Analytics5 months ago
Responsibilities
- Design and build scalable data platform components for batch and real-time data processing.
- Architect, develop, and operationalize large-scale data systems across ingestion, transformation, and serving layers.
- Build and manage reliable, scalable, and cost-efficient data pipelines.
- Develop reusable frameworks and tooling to accelerate data engineer and data scientist productivity.
- Use Python, Airflow, SQL, and cloud platforms to build production-grade data solutions.
- Optimize query performance and data models using columnar OLAP systems such as ClickHouse, Doris, and StarRocks.
- Implement streaming and near real-time data processing systems.
- Translate complex business requirements into scalable and efficient data platform solutions.
- Collaborate with cross-functional teams and provide technical leadership and mentorship.
- Drive architectural decisions by evaluating tradeoffs and selecting appropriate tools.
Requirements
- Bachelor's degree in Computer Science, Information Technology, or a similar discipline.
- 8+ years of professional experience in data engineering, backend systems, or distributed systems.
- Proven experience building scalable data platforms and large-scale data systems.
- Strong experience with ETL pipelines, data integration, and workflow orchestration systems such as Airflow or Temporal.
- Hands-on experience with Python and SQL and a strong understanding of data warehouse concepts.
- Experience with distributed OLTP/OLAP databases such as ClickHouse, PostgreSQL, Cassandra, or Elasticsearch.
- Knowledge of messaging and streaming systems such as Kafka.
- Experience with cloud platforms such as AWS or GCP and big data tools such as Spark.
- Strong understanding of columnar storage systems and query optimization techniques.
- Solid understanding of distributed-systems fundamentals and associated tradeoffs.
- Experience with containers and orchestration tools such as Docker and Kubernetes.
- Strong Linux fundamentals and system-level debugging skills.
- Familiarity with modern data architectures such as Lakehouse, including Iceberg, Hudi, or Delta, is a plus.
Tech Stack
Apache AirflowApache CassandraApache KafkaApache SparkAWSClickHouseDockerElasticsearchGoogle Cloud PlatformLinuxPostgreSQLPythonSQL
Categories
BackendData Engineering
About Arcana Analytics
Arcana enables institutional investors to understand their portfolio risks, decompose single stock & book performance, drill into crowding, and isolate their idiosyncratic differentiation. Built on our proprietary crowding, ownership, factor risk, and performance datasets. The company's investors include D1 Capital, Duquesne (Stan Druckenmiller), Tiger Global, Abstract Ventures, GoldenTree Asset Management, Ryan Roslansky (CEO LinkedIn), and Akshay Kothari (COO Notion), among others.