7 days ago
Base Salary
$180k - $190k/yr
Responsibilities
- Design scalable, secure, and high-performance AWS and Databricks solutions using AWS Networking, Kinesis, ElastiCache, Redshift, EKS, Cognito, and PaaS services.
- Develop and maintain ETL/ELT pipelines in Databricks using Python, PySpark, notebooks, jobs, Delta Lake tables, and Unity Catalog.
- Implement medallion architecture and optimize Spark jobs for performance, cost, scalability, and reliability.
- Provision and manage Databricks workspaces, clusters, jobs, storage, networking, IAM roles, permissions, and related AWS and Azure resources with Terraform.
- Write SQL for data transformation, validation, and analytics while ensuring data quality, lineage, security, and compliance.
- Implement CI/CD pipelines with Jenkins and GitHub Actions for notebooks, jobs, Delta Live Tables, and Terraform configurations.
- Integrate data from databases, APIs, streaming sources, and files into cloud storage and processing layers.
- Optimize cloud costs through autoscaling, spot instances, job scheduling, and efficient resource usage.
- Monitor pipeline performance, troubleshoot failures, and implement alerting and observability using Databricks, cloud monitoring, or third-party tools.
Requirements
- At least 15 years of experience in cloud architecture, including significant hands-on experience designing and implementing AWS and Databricks solutions.
- Advanced proficiency in AWS Networking and VPC design, AWS Kinesis, ElastiCache, Redshift, Lambda, S3, Glue, EKS, and Cognito.
- Expertise with Databricks, Delta Lake, Unity Catalog, Delta Live Tables, Python, PySpark, SQL, and Terraform.
- Experience with CI/CD tools including Jenkins and GitHub Actions.
- Preferred knowledge of Azure Data Lake, Data Factory, Synapse, and Key Vault.
- Preferred understanding of star/snowflake and dimensional data modeling, lakehouse architecture, Spark and Databricks performance tuning, data governance, lineage, and cloud compliance.
- A bachelor's degree in Computer Science, Information Technology, or a closely related discipline is a desired qualification.
- AWS Certified Solutions Architect – Professional and Databricks Certified Data Engineer Professional or Apache Spark certification are desired qualifications.
Benefits
- The role requires presence in St. Louis and attendance at the client office three days a week.
Categories
BackendData Engineering
About Brillio
Brillio is a privately held digital engineering and enterprise AI services firm that helps Fortune 1000 companies modernize products, data, and infrastructure. It combines consulting and managed services with an accelerator platform (ADAM) across cloud, AI, data engineering, customer experience, and infrastructure workstreams. Founded in 2014 and headquartered in Dallas, it operates delivery centers across North America, Europe, and Asia, with focus areas that include healthcare and life sciences.