Sr. Data Engineer

Job Desc
Core Responsibilities
• Design and build end-to-end data platforms using Microsoft Fabric
o Lakehouse, Warehouse, OneLake, Dataflows Gen2
• Develop and optimize Spark workloads using PySpark and SparkSQL
• Develop MLOps pipelines for Advanced Analytics & AI
• Build scalable ETL/ELT pipelines using:
o Azure Data Factory (ADF)
o MS Fabric Data pipeline
o Dataflow gen 2
o SSIS (on-prem, Azure-SSIS IR, and migration scenarios)
• Implement data modeling patterns:
o Medallion (Bronze / Silver / Gold)
o Dimensional modeling (Star/Snowflake)
o Different data file management experience – Parquet, JSON, XML
• Integrate Microsoft Purview for:
o Data cataloging & classification
o Automated data lineage (ADF, Fabric, SQL, ADLS)
• Enforce data security and access controls:
o RBAC, column-level security, masking
o Fabric & Purview policy alignment
• Optimize performance, reliability, and cost across Fabric capacities
• Implement CI/CD and IaC for data pipelines and governance artifacts
• Partner with security, compliance, and BI teams to ensure trusted data delivery

Required Technical Skills
Microsoft Fabric (Must-Have)
• Fabric Data Engineering workloads
• Lakehouse & Warehouse
• OneLake architecture
• Fabric pipelines & notebooks
• Capacity planning and performance optimization
• Advanced PySpark (joins, windows, UDFs, optimization)
• Strong SparkSQL
• Strong MLOps & Feature Engg.
• Partitioning strategies, shuffle tuning, caching
• Large-scale data processing (TB+)
Azure Data Platform
• Azure SQL Database / SQL Server
• Azure Data Factory (ADF)
• SSIS / Azure-SSIS Integration Runtime
• ADLS Gen2
Data Lineage & Security (Microsoft Purview)
• Purview data catalog & scanning
• Automated lineage across ADF, Fabric, SQL, ADLS
• Business glossary management
• Integration with Azure RBAC & security policies