Data Engineering & BI Lead (Azure + Databricks)/Remote

Apetan Consulting LLC is seeking a Data Engineering & BI Lead responsible for designing, building, and managing scalable data platforms and business intelligence solutions using Azure cloud services and Databricks. This role leads data engineering efforts, ensures data quality and governance, and enables data-driven decision-making through analytics and reporting.

Responsibilities:

• Design and implement data pipelines using Azure services and Databricks
• Build and maintain scalable data architectures (data lakes, data warehouses, lakehouse)
• Lead ETL/ELT development using tools like Azure Data Factory and Databricks
• Develop and optimize data models for analytics and reporting
• Oversee BI solutions (Power BI or similar) for dashboards and reporting
• Ensure data quality, integrity, and governance standards
• Monitor and optimize pipeline performance and costs
• Collaborate with business stakeholders to translate requirements into data solutions
• Lead and mentor a team of data engineers and BI developers
• Establish best practices for coding, testing, and deployment (CI/CD)

Requirements:

• Strong experience with Microsoft Azure data services
• Hands-on expertise in Databricks (PySpark, Spark SQL)
• Experience with Azure Data Factory, Azure Data Lake, and Synapse Analytics
• Proficiency in SQL and Python
• Solid understanding of data warehousing concepts and data modeling
• Experience with BI tools such as Power BI
• Knowledge of data governance, security, and compliance
• Strong leadership and stakeholder management skills
• Bachelor's or Master's degree in Computer Science, IT, or related field
• Azure certifications (e.g., Azure Data Engineer Associate)
• Experience with real-time/streaming data (e.g., Event Hubs, Kafka)
• Familiarity with DevOps practices and tools (Azure DevOps, Git)
• Experience in handling large-scale, distributed data systems