Bayforce is seeking a Senior Data Engineer to migrate enterprise data workloads from Azure Synapse Analytics to Microsoft Fabric. The role involves hands-on execution of migration tasks, collaborating with Data Architects and Engineers, and ensuring the implementation of Fabric-native solutions.
Responsibilities:
- Individually execute hands-on migration of enterprise data workloads from Azure Synapse Analytics to Microsoft Fabric, including pipelines, Spark notebooks, and SQL workloads
- Collaborate closely with Data Architects and fellow Data Engineers to translate target-state architecture into working Fabric solutions and ensure consistent implementation across migration efforts
- Review existing Synapse implementations (ADF/Synapse pipelines, Spark pools, SQL pools) and design, refactor, and implement Fabric-native equivalents using Lakehouse and Warehouse architectures (Bronze/Silver/Gold)
- Migrate data storage from ADLS Gen2 to OneLake, ensuring data parity, performance, security, and governance controls
- Build, optimize, and maintain end-to-end ELT/ETL pipelines using Microsoft Fabric Pipelines, Fabric notebooks, PySpark, SQL, and Azure Data Factory where required
- Develop and tune PySpark/Spark transformations for large-scale data processing, including incremental loads, partitioning strategies, joins, and file optimization
- Implement and support API-heavy ingestion frameworks (REST/SOAP), handling authentication, pagination, throttling, retries, and robust error handling
- Troubleshoot and resolve migration defects, performance regressions, and functional gaps between Synapse and Fabric
- Implement monitoring, logging, alerting, and operational runbooks for both migrated and net-new Fabric workloads
- Apply security, governance, and data quality standards across pipelines, including access controls, validation checks, lineage, and compliance requirements
Requirements:
- Bachelor's degree (or equivalent experience) in Computer Science, Engineering, or a related field
- 5+ years of hands-on data engineering experience in Azure cloud environments
- Strong, practical experience with Azure Synapse Analytics, including: Synapse Pipelines and/or Azure Data Factory, Synapse Spark pools, Dedicated and/or Serverless SQL pools
- Direct, hands-on experience migrating or modernizing data platforms, preferably from Synapse to Microsoft Fabric
- Hands-on Microsoft Fabric experience is REQUIRED, including: Fabric Lakehouse and Warehouse, OneLake, Fabric Pipelines, Fabric Notebooks, Dataflows Gen2
- Demonstrated experience rebuilding Azure Synapse patterns into Fabric-native solutions (not lift-and-shift)
- Advanced PySpark / Spark experience for complex transformations and performance tuning
- Strong SQL expertise and data warehousing concepts (dimensional modeling, incremental processing, data quality checks)
- Deep understanding of Data Lake, Lakehouse, and Warehouse architectures
- Proven ability to troubleshoot and optimize complex production data workflows