Key skills
Azure Data FactoryAzure DatabricksAzure Data Lake StoragePySparkApache SparkSQLT-SQLSpark SQLData pipelinesETL processesData warehousingDelta LakeData partitioningData indexingData optimizationCI/CD pipelinesDevOps practicesParquetJSONAvroStreaming technologiesEvent HubKafkaPythonStakeholder managementMachine LearningETLELTData EngineeringData WarehousingData LakeAnalyticsBIPower BISnowflakeDatabricksSparkAzureAgileScrumCI/CDStakeholder ManagementCommunication
About this role
Dice is seeking a skilled Azure Data Engineer with strong expertise in Azure Databricks, data pipelines, and big data processing. The ideal candidate will design, build, and optimize scalable data solutions on the Azure cloud platform, focusing on end-to-end data pipelines, ETL workflows, and data quality management.
Responsibilities:
- Design and develop end-to-end data pipelines using Azure Data Factory (ADF) and Azure Databricks
- Build and optimize ETL/ELT workflows for large-scale data processing
- Develop scalable data models and data lake architectures using Azure Data Lake Storage (ADLS)
- Work with structured and unstructured data from multiple sources
- Implement data transformation using PySpark / Spark SQL in Databricks
- Optimize performance of data pipelines and queries
- Ensure data quality, governance, and security
- Integrate data from various sources including APIs, databases, and streaming sources
- Collaborate with data scientists, analysts, and business teams
- Monitor, troubleshoot, and maintain data pipelines
Requirements:
- Strong experience with Azure Data Engineering services, including: Azure Data Factory (ADF), Azure Databricks, Azure Data Lake Storage (ADLS Gen2)
- Hands-on expertise in: PySpark / Apache Spark, SQL (T-SQL / Spark SQL)
- Experience in building data pipelines and ETL processes
- Knowledge of data warehousing concepts (Star/Snowflake schema)
- Experience with Delta Lake and data lake architectures
- Understanding of data partitioning, indexing, and optimization techniques
- Familiarity with CI/CD pipelines and DevOps practices
- Experience with data formats like Parquet, JSON, Avro
- Knowledge of streaming technologies (Event Hub, Kafka) is a plus
- Proficiency in Python for data processing
- Strong analytical and problem-solving skills
- Good communication and stakeholder management skills
- Ability to work independently and in a team environment
- Strong focus on data accuracy and quality
- Bachelor's degree in Computer Science, Engineering, or related field
- 12+ years of experience in data engineering or related roles
- At least 2+ years of hands-on experience with Azure Databricks
- Experience with Power BI / reporting tools
- Exposure to Machine Learning workflows
- Knowledge of Azure Synapse Analytics
- Experience working in Agile/Scrum environments