Senior Big Data Engineer
Qualys, Inc.1 hour ago
Pune, IndiaSenior
Responsibilities
- Lead the design, rollout, and evolution of scalable data platforms and pipelines.
- Prototype, develop, and support distributed SaaS security risk-prioritization systems processing billions of transactions per day.
- Provide technical leadership on architecture, design quality, system interdependencies, scalability, and performance.
- Drive technology exploration, roadmap development, cloud benchmarking, and trade-off reviews with product, professional services, and sales engineering teams.
- Define requirements and directions for developers and help decompose complex technical problems into straightforward solutions.
- Lead performance benchmarking, troubleshooting, and delivery of large-scale systems and infrastructure.
- Mentor engineers and provide technical training and career-development guidance across the engineering community.
- Design and implement secure big-data clusters that satisfy compliance and regulatory requirements.
Requirements
- Bachelor’s degree in computer science or equivalent.
- 5+ years of total professional experience.
- Experience building scalable, enterprise-grade big-data solutions with Java or Scala.
- At least 2 years of experience designing and architecting big-data solutions using Apache Spark.
- At least 3 years of experience working with engineering resources on innovation and at least 3 years of Kafka experience.
- At least 4 years of experience understanding big-data event-flow pipelines.
- At least 3 years of performance testing experience for large infrastructure.
- In-depth experience with search solutions such as Solr or Elasticsearch, data lakes, messaging queues, caching services, and scalable big-data or microservices architectures.
- Experience with RDBMS and NoSQL systems, including Oracle, Cassandra, Kafka, Redis, Hadoop, and related data architectures.
- Knowledge of Presto and Airflow, plus hands-on scripting and automation experience.
- Experience with Flink data streaming, rule engines, ML model engineering and deployment, and big-data services administration is preferred or advantageous.
- Experience with Agile management approaches, object-oriented modeling, Internet, UNIX, middleware, and database-related projects.
- Strong troubleshooting, performance benchmarking, architecture, technical leadership, mentoring, and training capabilities.
Tech Stack
Apache AirflowApache CassandraApache FlinkApache HadoopApache KafkaApache SparkElasticsearchJavaPrestoRedisScala
Categories
Data Engineering