13 hours ago
Mumbai, IndiaMid Level
Responsibilities
- Design, build, and operate the Java backend of a distributed job scheduling platform, including scheduling integrations, resource and cost tracking, state management, and failure recovery.
- Develop gRPC/protobuf APIs for Python, Java, and C++ clients.
- Build Kafka-based event-driven services for job lifecycle tracking, data lineage, and dataset orchestration.
- Work with Kubernetes at scale on pod scheduling, fair-share queueing, preemption, and multi-cluster placement.
- Instrument, monitor, operate, and troubleshoot production systems, including root-cause analysis of performance bottlenecks.
- Support and consult quantitative research users in the India office while collaborating with platform teams in Amsterdam, Chicago, and Sydney.
Requirements
- At least 3 years of professional Java development experience, including Java 17+, Maven or Gradle, concurrency, and JVM performance.
- At least 2 years of hands-on Kubernetes and Docker experience; experience building on the Kubernetes API is a strong plus.
- Comfort developing in Python and working across APIs used by Python-first research users.
- Solid understanding of distributed-systems fundamentals, including state machines, retries and idempotency, event-driven architectures, and consistency trade-offs.
- Experience with workload scheduling or batch and compute orchestration systems such as Kubernetes schedulers, Slurm, YARN, Kueue, Ray, or Airflow.
- Experience with messaging or streaming systems such as Kafka.
- Working knowledge of NoSQL and time-series or SQL databases such as MongoDB and PostgreSQL.
- Experience with gRPC/protobuf or similar RPC frameworks is a plus.
- Strong Linux fundamentals and scripting experience with Bash and Python.
- Experience with monitoring stacks such as Prometheus and Grafana, plus the ability to operate systems the candidate builds.
- Ability to troubleshoot across application code, Kubernetes, networking, and storage.
- Exposure to object storage such as S3 or high-performance data formats such as Parquet and Iceberg.
Tech Stack
Apache AirflowApache KafkaBashC++DockerGradleGrafanagRPCJavaKubernetesLinuxMavenMongoDBPostgreSQLPrometheusPythonYarn