2 days ago
Bengaluru, IndiaSenior
Responsibilities
- Design, build, optimize, and operate streaming and batch data pipelines for AI, machine learning, analytics, and operational use cases.
- Manage Apache Kafka clusters and ecosystem components, including Kafka Connect, KSQL, AWS MSK, monitoring, maintenance, scaling, and incident response.
- Operate Redis, PostgreSQL, MySQL, MongoDB, Redshift, and Snowflake environments, including tuning, replication, high availability/disaster recovery, upgrades, and capacity planning.
- Develop Python and Bash automation, CI/CD pipelines, and infrastructure-as-code to reduce operational toil and improve reliability.
- Containerize and orchestrate workloads with Docker and Kubernetes, including participation in a Kafka and related-services migration.
- Manage AWS infrastructure and services, including EC2, VPC, IAM, MSK, ElastiCache, CloudWatch, Athena, Glue, Redshift, and S3-based data lake patterns.
- Maintain observability through metrics, logs, and tracing; participate in on-call incident response and root cause analysis.
- Support cloud migration, data security, secrets management, least-privilege access, and compliance practices across data stores and infrastructure.
Requirements
- At least 5 years of overall engineering experience spanning data pipelines, database operations, and infrastructure.
- Production experience with Kafka or another distributed streaming platform such as Kinesis or Pulsar.
- Operational experience with at least two of Redis, PostgreSQL, MySQL, MongoDB, Redshift, or Snowflake.
- Experience building or maintaining ETL/ELT processes, batch jobs, or streaming pipelines.
- Strong Python and/or Bash scripting and automation experience.
- Experience developing CI/CD pipelines and using infrastructure-as-code, with Terraform preferred.
- Solid Linux fundamentals, including networking, log analysis, troubleshooting, and process, memory, and disk management.
- Cloud engineering experience, preferably with AWS services including EC2, VPC, IAM, MSK, ElastiCache, and CloudWatch.
- Familiarity with observability practices and incident response workflows.
- Understanding of distributed-systems trade-offs such as availability, consistency, partitions, and backpressure.
- Strong communication skills and ability to present platform status and technical decisions to non-technical stakeholders.
- Willingness to participate in on-call rotation and provide occasional weekend support.
- Preferred experience includes Kafka Streams, Apache Flink, Apache Iceberg, Delta Lake, Apache Hudi, AWS data services, Snowflake, data quality, data lineage, schema registries, Kubernetes, Helm, VMware/KVM, SAN/NAS storage, PCI-DSS, HashiCorp Vault, and AWS Secrets Manager.
Benefits
- Broad ownership across the data platform and direct impact on global payment infrastructure.
- Hands-on participation in a Kafka-to-Kubernetes infrastructure modernization project.
- Mentorship from senior engineers and platform architects with production experience.
- Funded certification paths for Confluent, AWS, CKA, and dbt, plus a budget for courses and technical conferences.
- Hybrid/remote working flexibility with a collaborative team culture.
- Competitive compensation with performance-based incentives.
- In-office work culture is emphasized, with physical co-location described as beneficial to career development and business results.
Tech Stack
Amazon RedshiftAnsibleApache FlinkApache KafkaAWSBashdbtDockerHelmKubernetesMicrosoft SQL ServerMongoDBMySQLPostgreSQLPythonRedisSnowflakeTerraform
Categories
Data EngineeringDevOps
