4 hours ago
Responsibilities
- Design, develop, and maintain robust Python, PySpark, and SQL ingestion, transformation, and enrichment pipelines.
- Write and optimize complex SQL queries, analytical UDFs, stored procedures, and window functions for aggregation and reporting.
- Collaborate with data architects, data scientists, and business analysts to translate requirements into technical specifications.
- Perform unit testing, integration testing, code reviews, production support, and bug fixing.
- Maintain CI/CD pipelines using Git, Jenkins, and Docker for automated build, testing, and deployment.
- Monitor production workloads and troubleshoot performance bottlenecks, memory issues, and job failures.
- Document data lineage, pipeline designs, and operational runbooks in Confluence and SharePoint.
- Mentor junior engineers and promote Python, Spark optimization, and data-engineering best practices.
- Evaluate emerging technologies, contribute to AI/ML innovation, and deliver proof-of-concepts for CEFS.
- Participate in Agile ceremonies and backlog grooming.
Requirements
- 5–7 years of experience as a Python Developer working with Spark or PySpark.
- Strong hands-on experience with Python, PySpark, advanced SQL, and ETL in a data engineering environment.
- Experience with PySpark DataFrames, Spark SQL, Structured Streaming, partitioning, caching, broadcast joins, and performance tuning.
- Knowledge of Python packages including NumPy and pandas for data cleaning, wrangling, analysis, and visualization.
- Experience with MS-SQL or Oracle, Linux/Unix environments, testing phases, documentation, and production support.
- Experience with build tools such as Maven and DevOps tools such as Bitbucket and Jenkins.
- Knowledge of object-oriented programming, APIs, design patterns, development principles, and multi-integration applications.
- Knowledge of front-end technologies, preferably Flask, is desirable.
- Bachelor’s degree or equivalent.
- Ability to collaborate, simplify complex topics, deliver results, maintain attention to detail, and learn diverse technologies.
Benefits
- Innovation-focused work culture with ongoing learning opportunities.
- Flat, non-hierarchical structure with opportunities to work directly with senior partners and clients.
- Diverse, inclusive, and meritocratic culture.
- Work location: Chennai.
About Capco
Capco, a Wipro company, is a global technology and management consultancy specializing in driving digital transformation in the financial services industry. With a growing client portfolio comprising of over 100 global organizations, Capco operates at the intersection of business and technology by combining innovative thinking with unrivalled industry knowledge to deliver end-to-end data-driven solutions and fast-track digital initiatives for banking and payments, capital markets, wealth and asset management, insurance, and the energy sector. Capco’s cutting-edge ingenuity is brought to life through its Innovation Labs and award-winning Be Yourself At Work culture and diverse talent.