Lead Software Engineer - Big Data
Financial Industry Regulatory Authority, Inc. (FINRA)3 months ago
Rockville, MD, USA or Washington, DC, USAStaff+
Base Salary
$114k - $249k/yr
Responsibilities
- Own the development and continuous quality improvement of software products throughout the software development life cycle.
- Design and develop scalable, reliable, distributed data-processing solutions and enterprise applications.
- Process petabyte-scale datasets with Hadoop, Spark, Hive, and Trino and diagnose production scalability, resource, data-skew, and job-failure issues.
- Tune Apache Spark and SQL workloads using partitioning, caching, broadcast joins, query optimization, and related techniques.
- Build and maintain software development environments, DevOps practices, CI/CD pipelines, and infrastructure-as-code solutions.
- Improve data storage, data modeling, application performance, security, test automation, and unit-testing practices.
- Analyze software engineering trends and test-environment tools and communicate actionable integration recommendations.
- Mentor, coach, and train software engineers while partnering across teams to remove barriers and achieve shared priorities.
- Lead or support AI adoption, technology transformation, change-management, and workflow-optimization initiatives.
Requirements
- Bachelor’s degree in Computer Science, Information Systems, or a related discipline, with at least 7 years of related experience, or equivalent training and/or work experience.
- Extensive hands-on experience with Hadoop, Spark, Hive, and Trino for petabyte-scale data processing.
- Deep knowledge of Spark architecture and performance tuning, including executors, tasks, stages, DAG execution, partitioning, caching, and broadcast joins.
- Advanced SQL proficiency covering window functions, multi-table joins, aggregations, query optimization, NULLs, duplicates, and ordering.
- Strong experience with AWS data and cloud services, including S3, EMR, Glue, Lambda, Athena, and EKS.
- Proficiency in Python or Scala, including functional programming, collections, concurrency, memory management, and performant modular code.
- Strong system design, object-oriented development, design-pattern, data-storage, database, data-modeling, and performance-optimization experience.
- Experience with relational and NoSQL systems, DevOps, CI/CD, infrastructure as code, software security, test automation, and unit testing.
- Hands-on experience with GitHub Copilot, Q Developer, ChatGPT, Claude, prompt engineering, and AI workflow design.
- Experience with Scrum, Kanban, continuous improvement, team leadership, AI adoption, technology transformation, and workflow optimization.
- Ability to interpret AI-generated insights and data-driven metrics and translate them into technical improvements and business value.
- Strong collaboration, communication, independent judgment, coaching, and cross-functional influence skills.
Benefits
- Hybrid work environment with defined in-person presence requirements; occasional travel and extended hours may be required.
- Base salary ranges from $114,200 to $207,200 in CO/FL/TX, $125,900 to $228,000 in IL/PA, $131,200 to $238,300 in MA/MD/VA/Washington, DC, and $131,200 to $248,700 in NY/NJ.
- Employees may be eligible for a discretionary bonus, and non-exempt employees may receive overtime pay where applicable.
- Comprehensive health, dental, vision, life, disability, long-term-care, legal, and business-travel insurance options.
- Immediate 401(k) participation and vesting with company match, plus eligibility for an additional FINRA-funded retirement contribution.
- Tuition reimbursement, commuter benefits, adoption assistance, backup family care, surrogacy benefits, employee assistance, and wellness programs.
- Fifteen paid time-off days, five personal days, nine sick days, two volunteer service days, paid holidays, and other paid leave including parental leave after 90 days of employment.
Tech Stack
Categories
Data Engineering