Lead Data Engineer (AWS Data Platform)
NorthBay Solutions4 months ago
Karachi, PakistanStaff+
Responsibilities
- Design, develop, and maintain scalable data pipelines using PySpark, AWS Glue, and Amazon EMR.
- Build and optimize data ingestion, transformation, and processing frameworks for structured and semi-structured data.
- Develop and maintain enterprise data warehouse solutions using Amazon Redshift.
- Write complex SQL queries, stored procedures, and data transformations for analytics and reporting.
- Implement ETL/ELT processes across multiple systems and platforms.
- Perform performance tuning and optimization of Spark jobs, ETL pipelines, SQL queries, and Redshift workloads.
- Ensure data quality, integrity, security, and governance across data platforms.
- Troubleshoot production issues and perform root cause analysis for data-related incidents.
- Collaborate with business stakeholders, analysts, architects, and engineering teams on data requirements.
- Participate in code reviews, technical design discussions, and best-practice implementation.
- Monitor data pipelines and identify opportunities for performance improvement and automation.
- Create and maintain technical documentation, data models, and operational procedures.
Requirements
- 5 to 8+ years of experience in data engineering and data warehousing.
- Strong hands-on experience with PySpark, AWS Glue, Amazon EMR, and Amazon Redshift.
- Excellent SQL development and query optimization skills.
- Strong understanding of data warehousing, dimensional modeling, and ETL/ELT processes.
- Experience tuning Spark jobs, SQL queries, ETL pipelines, and data warehouse workloads.
- Experience handling large-scale datasets and distributed data processing.
- Strong debugging, troubleshooting, and analytical skills.
- Bachelor's degree in Computer Science, Software Engineering, Information Technology, or a related field.
- Experience with AWS services such as S3, IAM, CloudWatch, Lambda, and Step Functions is preferred.
- Knowledge of CI/CD pipelines, DevOps practices, workflow orchestration, and data governance, security, and compliance is preferred.
- Relevant AWS certifications are advantageous.
- Strong communication, stakeholder management, independent work, problem-solving, collaboration, and ownership skills are required.
Benefits
- Hybrid work arrangement in Pakistan.
- Candidates must be available during UAE business hours and follow UAE public holidays.
- The role involves collaboration with a UAE-based customer and distributed teams.
Tech Stack
Categories
Data Engineering
About NorthBay Solutions
NorthBay Solutions is a privately held IT services firm that helps enterprises modernize on AWS through cloud migration, data engineering, and AI/ML solutions. An AWS Premier Consulting Partner founded in 2010 and headquartered in Andover, MA, it also partners with Snowflake and provides managed services, DevOps enablement, and application development. The company operates distributed teams across the US, Canada, UAE, India, and Pakistan.