Senior Data Engineer
Arcana Analytics1 year ago
Bengaluru, IndiaMid Level / Senior
Responsibilities
- Develop and implement real-time data ingestion and processing systems.
- Design, build, and operationalize large-scale enterprise data solutions and applications.
- Create and manage production data pipelines from ingestion to consumption within a big data architecture using PySpark.
- Use Python, Airflow, SQL, and cloud platforms to build robust data engineering solutions.
- Translate complex business challenges into scalable and efficient technical solutions.
- Own the full lifecycle of solution implementation in collaboration with the data engineering team.
- Develop and improve compute- and I/O-intensive applications for performance and reliability.
Requirements
- Bachelor's degree in Computer Science, Information Technology, or a similar discipline.
- 3–8 years of professional experience in data engineering or related fields.
- Experience with ETL processes, data integration, large-scale datasets, and PySpark.
- Strong understanding of ETL, near- and real-time streaming, data structures, and workflow management.
- Strong knowledge of Python, Airflow, SQL, and cloud platforms.
- Experience with version control tools such as Git and GitHub.
- Proficiency in SQL and data warehouse concepts.
- Experience with SQL or NoSQL databases such as Cassandra, MongoDB, or HBase.
- Knowledge of AWS technologies including EMR, Redshift, Kinesis, Lambda, Glue, S3, IAM, and CloudWatch.
- Knowledge of big data tools including Hadoop, EMR, Hive, and Sqoop.
Tech Stack
Categories
Data Engineering
About Arcana Analytics
Arcana Analytics builds a portfolio intelligence platform for hedge funds and asset managers, combining portfolio data with analytics on performance attribution, factor risk, crowding, and ownership using proprietary datasets. It sells enterprise software and data subscriptions used for portfolio construction, real-time risk, and research workflows by institutional investors in global markets, including APAC. Founded in 2022 and headquartered in New York, the company is privately held and raised a Series B in 2026.