REMOTE - Lead MLOps Engineer (Databricks)
State Farm2 months ago
Bloomington, IL, USA +3 moreStaff+
Base Salary
$100k - $200k/yr
Responsibilities
- Design and operate Bronze-to-Silver-to-Gold medallion data architecture with data quality, schema management, and Delta Lake tuning.
- Optimize Spark queries, partitioning, caching, and Databricks cluster sizing for application performance.
- Own Unity Catalog schemas, access controls, project tagging, and lineage-driven issue ticketing.
- Manage model lifecycles in MLflow from experimentation through production, retraining, drift detection, and retirement.
- Track, alert on, and optimize Databricks spending at the project level.
- Define production handoff standards including runbooks, self-healing, governance, FinOps controls, and incident targets.
- Create reusable templates, patterns, and standards for future projects.
- Integrate agentic AI into validation, pipeline generation, testing, monitoring, and documentation.
- Set technical direction, mentor engineers, and collaborate on architecture, code reviews, and debugging.
Requirements
- At least one year of production Databricks experience at the application layer, including medallion architecture, Delta Lake, Unity Catalog, and PySpark.
- Hands-on experience operating production ML systems, including incident response, root-cause analysis, drift detection, and silent-degradation prevention.
- Daily practical use of GenAI and agentic tools, with strong command-line comfort and an automation mindset.
- Hands-on MLflow experience covering registry, experiment tracking, deployment automation, retraining pipelines, and drift detection.
- Experience tracking and optimizing Databricks costs and managing project-level compute budgets.
- Experience building or operating end-to-end data lineage and observability systems with automated issue creation.
- Experience preparing systems for operations teams using runbooks, self-healing patterns, incident targets, and governance checklists.
- Production-grade Python, PySpark, SQL, and AWS experience, plus CI/CD for data and ML pipelines and testing strategies for non-deterministic systems.
- Familiarity with Hidden Technical Debt in Machine Learning Systems is expected.
- Experience with Omnigent, Databricks' open-source agent meta-harness, is preferred.
Benefits
- Potential starting base salary of $100,000-$200,000 and incentive pay of up to 15% of base salary.
- Remote work is available outside a 50-mile radius of listed hubs; hybrid work applies within 50 miles of Bloomington, IL, Dunwoody, GA, Richardson, TX, or Tempe, AZ.
- Health, dental, vision, telemedicine, mental health, and wellbeing benefits, with State Farm paying most healthcare premiums.
- Training programs, tuition assistance, employee resource groups, and mentoring.
- Fertility, IVF, adoption assistance, college coaching, discounts, financial workshops, and financial coaching.
- Time off including up to 20 days initially, parental leave, paid holidays, life leave, bereavement leave, and community service or education support days.
- Matching gifts, community grants, and an employee assistance fund.
- 401(k) plan with company contributions of up to 7% of salary and access to financial advisors.
- Standard 38:45-hour work week.
Tech Stack
Categories
Data EngineeringML Engineering
About State Farm
State Farm is a U.S. mutual insurance group that sells auto, home, renters, life, health, and small-business coverage, plus select financial services. It markets policies primarily through a large network of local agents and earns revenue from premiums and investing customer reserves. Founded in 1922 and headquartered in Bloomington, Illinois, it is among the largest personal lines insurers in the United States.