Senior Data Engineer
Sumo Logic4 days ago
Noida, IndiaSenior
Responsibilities
- Design, develop, and optimize scalable data pipelines using Databricks, Apache Spark, PySpark, Python, SQL, and Delta Lake.
- Implement Bronze, Silver, and Gold medallion architecture layers for trusted, analytics-ready datasets.
- Build and maintain integrations with enterprise applications, REST APIs, databases, and other source systems.
- Develop reusable ingestion frameworks and reliable batch and near-real-time pipelines with incremental loads, CDC, schema evolution, and error recovery.
- Configure Databricks Workflows and implement ingestion and transformation pipelines using Auto Loader and Lakeflow Spark Declarative Pipelines.
- Optimize Spark and Delta Lake workloads through execution-plan analysis, join optimization, data-skew handling, partitioning, and SQL tuning.
- Implement Unity Catalog governance, including access controls, lineage, auditing, masking, and row-level security.
- Maintain data workflows on Amazon Redshift and Amazon EC2 and collaborate on IAM permissions, S3 access, connectivity, deployments, and operational issues.
- Follow version control, code review, automated testing, CI/CD, and documentation practices.
- Translate requirements from analytics and business stakeholders into trusted, documented datasets.
- Provide technical guidance and improve automation, engineering standards, platform reliability, and resource utilization.
Requirements
- At least 5 years of experience in data engineering or a related technical role, including production data pipeline development and support.
- Strong hands-on Databricks experience, including developing, troubleshooting, and optimizing production workloads.
- Strong proficiency in Python or PySpark, SQL, and Apache Spark.
- Practical experience with Delta Lake, including table management, incremental processing, and performance optimization.
- Experience with data architecture and modeling, medallion architecture, and analytics-ready datasets.
- Experience with Databricks Workflows or equivalent orchestration solutions.
- Proven experience building ingestion pipelines and integrating REST APIs, databases, and enterprise applications.
- Understanding of retries, idempotency, schema evolution, backfills, and failure recovery.
- Working knowledge of AWS infrastructure, IAM roles and policies, S3 storage and access management, and secure connectivity.
- Experience with Git, GitHub, code reviews, automated testing, and CI/CD practices.
- Ability to independently design, build, deploy, and maintain production data solutions.
- Strong communication and collaboration skills with analytics, engineering, IT/cloud, security, and business stakeholders.
- Desirable experience with Lakeflow Spark Declarative Pipelines, formerly Delta Live Tables, Auto Loader, advanced incremental ingestion, Unity Catalog governance, Databricks Genie, Genie Agents, Genie Code, AI/ML, data science, LLMs, contextualization, and Amazon Bedrock.
Benefits
- Hybrid work arrangement in Noida.
Tech Stack
Categories
Data Engineering
About Sumo Logic
Sumo Logic builds a cloud-native SaaS platform for log management, security analytics (SIEM), and observability used by DevOps and security teams to monitor and troubleshoot applications and infrastructure. Founded in 2010 and headquartered in Redwood City, California, it was acquired by Francisco Partners and taken private in 2023. The business is subscription-based and supports cloud and hybrid environments.