Lead Azure Databricks Data Engineer

Role Overview

• Provide technical leadership in data architecture and engineering, collaborating with solution stakeholders to align data solutions with business objectives
• Mentor less experienced data engineers and team members, fostering a collaborative and innovative work environment
• Explore and implement new technologies and practices in data engineering to continually improve our data processes and infrastructure
• Manage and enhance complex SSIS packages for data integration and ETL processes, along with SSRS reports for business intelligence
• Design, build, and maintain scalable and efficient data pipelines using Azure Data Factory, implementing both batch and real-time data integration processes
• Develop and optimize Databricks notebooks for big data processing and analytics in Azure Databricks environments
• Implement and manage data storage solutions using Delta Parquet files in Azure Data Lake Storage, optimizing for query performance and cost-efficiency
• Integrate and configure Profisee Master Data Management tools within Azure, focusing on data governance and quality
• Set up and manage Change Data Capture processes for real-time data synchronization in Azure data platforms
• Implement data cataloging, classification, and lineage using Azure Purview, developing data governance policies, and ensuring compliance
• Enhance and support Tableau reporting solutions, developing complex dashboards, and integrating with Azure data services
• Drive automation and process optimization using Azure services and scripting and implement CI/CD pipelines using Azure DevOps
• Lead SQL Server database optimization and performance tuning efforts
• Lead, Coach and Mentor junior staff on projects

Requirements

• Bachelor’s degree required, Master’s degree preferred; Computer science, Information Technology or Engineering is preferred
• 10+ years of experience in Data Engineering with a Bachelor’s degree, or 8 years of experience with a Master’s degree is preferred
• 6+ years of experience with Databricks
• Subject Matter Expert in SQL Server Management, Azure Data Factory Development, Databricks Engineering, Delta Parquet File Management, and SQL Server Integration Services
• Proficiency in Profisee MDM Integration, Change Data Capture implementation, and data quality and governance with Azure Purview
• Proven experience in Tableau and BI Reporting, and strong skills in Automation and Process Skills

Tech Stack

• Azure
• ETL
• SQL
• SSIS
• Tableau

Benefits

• Paid time off for company holidays, vacation, sick and personal days
• Paid parental leave
• Mental health services