3 months ago
Bengaluru, IndiaStaff+
Responsibilities
- Design architectures, build proofs of concept, evaluate trade-offs, and deliver new data services and migrations.
- Build and maintain scalable real-time event collection and processing systems.
- Develop data pipelines and infrastructure supporting AI model training, inference, autonomous agents, and the MCP server.
- Detect, alert on, and resolve data quality issues while improving data trust and availability.
- Enhance ingestion, storage, and processing systems using current engineering standards and optimize their unit economics.
- Own services and the complete development lifecycle from ideation through deployment and continuous improvement.
- Establish monitoring, alerting, and operational procedures for mission-critical data systems.
- Collaborate cross-functionally, maintain technical documentation and architecture diagrams, and support stakeholder access to data.
- Govern company-wide data assets, audit data posture, reduce time to insight, and support data-informed business decisions.
- Share knowledge, improve team productivity, and mentor junior and senior engineers.
Requirements
- 8+ years of progressive professional experience in data engineering, including designing, building, and operating large-scale, high-throughput data platforms.
- Experience with extreme-scale systems handling hundreds of millions of events or requests per minute, billions of daily events, and petabytes of storage.
- Strong proficiency in at least one of Python, Java, Scala, or Go.
- Deep practical experience with distributed data processing frameworks such as Apache Spark or Apache Flink for batch and streaming workloads.
- Experience building data pipelines for machine learning model training, inference, and feature stores, with familiarity with AI agent data needs.
- Extensive experience with high-throughput messaging and streaming platforms such as Apache Kafka or Kinesis.
- Hands-on expertise with AWS, GCP, or Azure, preferably AWS and its data services.
- Strong experience with data warehousing, data lakes, ETL/ELT orchestration, relational and NoSQL databases, and advanced data modeling.
- Strong SQL skills and experience with databases such as PostgreSQL, MySQL, Cassandra, or DynamoDB.
- Exceptional analytical, debugging, distributed-systems optimization, communication, collaboration, technical leadership, and mentoring abilities.
- Experience in consumer-facing or streaming industries is preferred.
Tech Stack
Amazon DynamoDBAmazon RedshiftApache AirflowApache CassandraApache FlinkApache KafkaApache SparkAWSAzureGoGoogle BigQueryGoogle Cloud PlatformJavaMySQLPostgreSQLPythonScalaSnowflakeSQL
Categories
Data Engineering
About Hotstar
We’ve moved! 🌟 Follow us on @JioHotstar for all the latest stories, updates & more.