Project Delivery Specialist - Lead Databricks Engineer - Data Lakes

Inclusively is a digital tech platform that empowers job seekers with disabilities, caregivers, and veterans. They are seeking a Project Delivery Specialist - Lead Databricks Engineer to lead the end-to-end delivery of data lake workloads on Databricks, architecting and implementing pipelines, and establishing engineering standards.

Responsibilities:

• Lead end-to-end delivery of data lake/lakehouse workloads on Databricks: ingestion, processing, curation, and consumption layers
• Architect and implement pipelines using Spark (PySpark/Scala), Delta Lake, and orchestration tooling (e.g., Databricks Workflows, ADF/Airflow—as applicable)
• Define data modeling patterns (bronze/silver/gold, dimensional/denormalized, CDC handling) and optimize for analytics use cases
• Establish engineering standards: repo structure, coding conventions, branching strategy, documentation, and reusable frameworks
• Implement data quality and validation controls (reconciliation, anomaly detection, schema enforcement, expectations)
• Drive performance tuning: partitioning, file sizing, Z-ORDER, caching, cluster sizing, job parallelism, and query optimization
• Partner with governance/security to implement access controls (e.g., Unity Catalog, RBAC/ABAC), PII handling, encryption, and audit logging. Build/maintain CI/CD (continuous integration/continuous delivery) for notebooks/jobs (Databricks Repos, Git integration; IaC where applicable)
• Monitor and improve operational reliability: observability dashboards, alerting, runbooks, incident response support
• Manage dependencies across teams (source system owners, platform/infrastructure, BI, data science) and provide delivery estimates and plans
• Mentor engineers, conduct code reviews, and lead technical design sessions
• Communicate regularly with Engagement Managers (Directors), project team members, and representatives from various functional and / or technical teams, including escalating any matters that require additional attention and consideration from engagement management
• Independently and collaboratively lead client engagement workstreams focused on improvement, optimization, and transformation of processes including implementing leading practice workflows, addressing deficits in quality, and driving operational outcomes

Requirements:

• 7+ years experience with Apache Spark, Databricks SQL, Python, ETL, Lakeflow, Cloud Orchestration
• 7+ years in data engineering with strong hands-on Databricks experience in production environments
• Advanced skills in Spark and PySpark/Scala, with strong SQL capabilities
• Deep experience with Delta Lake and lakehouse/data lake design patterns
• Experience building batch and/or streaming pipelines; familiarity with CDC concepts
• Proven technical leadership (leading squads, setting standards, design ownership)
• Experience with cloud storage and security fundamentals (object storage, networking basics, IAM)
• Bachelor's degree, preferably in Computer Science, Information Technology, Computer Engineering, or related IT discipline; or equivalent experience
• Limited immigration sponsorship may be available
• Ability to travel 10%, on average, based on the work you do and the clients and industries/sectors you serve