3 days ago
Bengaluru, IndiaStaff+
Responsibilities
- Design architectures, build proofs of concept, and evaluate trade-offs for new services and migrations.
- Build and maintain scalable, accurate, clean, real-time event collection and data pipelines.
- Develop data infrastructure supporting AI model training, inference, feature stores, AI agents, and the team’s MCP server.
- Detect, alert on, and resolve data quality issues while improving data trust and availability.
- Enhance existing systems using current tooling and engineering standards, while optimizing ingestion, storage, and processing costs.
- Own the complete development lifecycle and services within the workstream, from ideation through deployment and continuous improvement.
- Establish monitoring, alerting, and operational procedures for mission-critical data systems.
- Collaborate cross-functionally, document architectures, and break high-level designs into services and tasks.
- Govern company-wide data assets, audit data posture, reduce time to insight, and make data accessible to stakeholders.
- Mentor engineers, share knowledge through blogs and articles, and improve team productivity.
Requirements
- 8+ years of progressive professional experience in data engineering.
- Proven experience designing, building, and operating large-scale, high-throughput data platforms, ideally in consumer-facing or streaming environments.
- Experience with systems handling hundreds of millions of events or requests per minute, billions of daily events, and petabytes of storage.
- Strong proficiency in at least one of Python, Java, Scala, or Go.
- Deep practical experience with distributed processing frameworks such as Apache Spark or Apache Flink for batch and streaming workloads.
- Experience building data pipelines for machine learning training, inference, and feature stores, with familiarity with AI agent data requirements.
- Extensive experience with messaging and streaming platforms such as Apache Kafka or Kinesis.
- Hands-on expertise with AWS, GCP, or Azure, preferably AWS and its data services.
- Strong experience with data warehousing and data lake architectures, including Snowflake, BigQuery, Redshift, Delta Lake, Iceberg, or Hudi.
- Expertise building, optimizing, and orchestrating ETL/ELT pipelines with Apache Airflow, Prefect, or comparable workflow systems.
- Strong SQL skills and experience with relational databases such as PostgreSQL and MySQL and NoSQL databases such as Cassandra and DynamoDB.
- Advanced data modeling skills, including dimensional, Kimball, and data vault approaches.
- Exceptional analytical, debugging, optimization, communication, collaboration, mentoring, and technical leadership abilities.
Tech Stack
Amazon DynamoDBAmazon RedshiftApache AirflowApache CassandraApache FlinkApache KafkaApache SparkAWSAzureGoGoogle BigQueryGoogle Cloud PlatformJavaMySQLPostgreSQLPythonScalaSnowflakeSQL
Categories
Data Engineering
About Hotstar
We’ve moved! 🌟 Follow us on @JioHotstar for all the latest stories, updates & more.