Inclusively is a digital tech platform that empowers job seekers with disabilities, caregivers, and veterans. They are seeking a Project Delivery Specialist - Lead Databricks Engineer to lead the end-to-end delivery of data lake workloads on Databricks, architecting and implementing pipelines, and establishing engineering standards.
Responsibilities:
- Lead end-to-end delivery of data lake/lakehouse workloads on Databricks: ingestion, processing, curation, and consumption layers
- Architect and implement pipelines using Spark (PySpark/Scala), Delta Lake, and orchestration tooling (e.g., Databricks Workflows, ADF/Airflow—as applicable)
- Define data modeling patterns (bronze/silver/gold, dimensional/denormalized, CDC handling) and optimize for analytics use cases
- Establish engineering standards: repo structure, coding conventions, branching strategy, documentation, and reusable frameworks
- Implement data quality and validation controls (reconciliation, anomaly detection, schema enforcement, expectations)
- Drive performance tuning: partitioning, file sizing, Z-ORDER, caching, cluster sizing, job parallelism, and query optimization
- Partner with governance/security to implement access controls (e.g., Unity Catalog, RBAC/ABAC), PII handling, encryption, and audit logging. Build/maintain CI/CD (continuous integration/continuous delivery) for notebooks/jobs (Databricks Repos, Git integration; IaC where applicable)
- Monitor and improve operational reliability: observability dashboards, alerting, runbooks, incident response support
- Manage dependencies across teams (source system owners, platform/infrastructure, BI, data science) and provide delivery estimates and plans
- Mentor engineers, conduct code reviews, and lead technical design sessions
- Communicate regularly with Engagement Managers (Directors), project team members, and representatives from various functional and / or technical teams, including escalating any matters that require additional attention and consideration from engagement management
- Independently and collaboratively lead client engagement workstreams focused on improvement, optimization, and transformation of processes including implementing leading practice workflows, addressing deficits in quality, and driving operational outcomes
Requirements:
- 7+ years experience with Apache Spark, Databricks SQL, Python, ETL, Lakeflow, Cloud Orchestration
- 7+ years in data engineering with strong hands-on Databricks experience in production environments
- Advanced skills in Spark and PySpark/Scala, with strong SQL capabilities
- Deep experience with Delta Lake and lakehouse/data lake design patterns
- Experience building batch and/or streaming pipelines; familiarity with CDC concepts
- Proven technical leadership (leading squads, setting standards, design ownership)
- Experience with cloud storage and security fundamentals (object storage, networking basics, IAM)
- Bachelor's degree, preferably in Computer Science, Information Technology, Computer Engineering, or related IT discipline; or equivalent experience
- Limited immigration sponsorship may be available
- Ability to travel 10%, on average, based on the work you do and the clients and industries/sectors you serve