Staff Big Data Engineer
Qualys, Inc.3 months ago
Pune, IndiaStaff+
Responsibilities
- Define the technical vision, architecture, and long-term strategy for the data platform and pipeline ecosystem.
- Design, build, and maintain scalable, highly available data processing systems using Apache Spark, Apache Kafka, and distributed data technologies.
- Lead architecture and design decisions across multiple engineering teams with emphasis on scalability, reliability, performance, security, and cost efficiency.
- Implement event-driven, streaming, and large-scale data processing architectures for real-time and batch workloads.
- Troubleshoot and optimize performance bottlenecks across data pipelines, distributed systems, and platform infrastructure.
- Partner with Product Management, Professional Services, and Sales Engineering to evaluate technical solutions and architecture trade-offs.
- Establish engineering standards, architectural guidelines, documentation practices, and operational best practices.
- Research and recommend technologies that improve platform scalability, performance, and operational efficiency.
- Mentor engineers in distributed systems design, performance optimization, and big data technologies.
- Support architecture reviews, technical governance initiatives, and engineering excellence programs.
Requirements
- At least 12 years of experience in software engineering, data engineering, distributed systems engineering, or big data platform development.
- At least 6 years of hands-on experience designing, developing, and optimizing Apache Spark-based data processing solutions.
- At least 6 years of experience building and operating large-scale data pipelines processing billions of events or transactions daily.
- At least 4 years of experience administering and operating Apache Kafka in production.
- At least 4 years of experience designing and implementing event-driven or streaming data architectures.
- Strong expertise in distributed data processing systems, data lake architectures, and large-scale platform design.
- Experience delivering highly available, scalable, and fault-tolerant distributed systems in production.
- Experience leading architecture initiatives and influencing technical direction across multiple engineering teams.
- Bachelor’s degree in Computer Science, Computer Engineering, Information Technology, or a related technical discipline.
- Experience with high-volume, multi-tenant enterprise platform environments is preferred.
- Proficiency in Scala, Java, or Python for data processing applications is preferred.
- Experience mentoring junior or mid-level engineering team members is preferred.
Tech Stack
Categories
Data Engineering
About Qualys, Inc.
Qualys builds a cloud-based security and compliance platform used by enterprises to automate vulnerability management, IT asset visibility, web application scanning, and policy/PCI compliance across on‑prem and cloud environments. Delivered as subscription SaaS, its Enterprise TruRisk Platform uses a single agent to provide continuous security telemetry for endpoints, servers, containers, and clouds. Founded in 1999 and headquartered in Foster City, California, Qualys is a public company trading on NASDAQ (QLYS).