ETL / ELT Architect (Azure Data Engineering)

Connect Tech+Talent is seeking an ETL / ELT Architect specializing in Azure Data Engineering. This role involves leading the design and governance of scalable data pipelines on Microsoft Azure, focusing on architecture standards and performance optimization.

Responsibilities:

• Define and govern Medallion Architecture (Bronze, Silver, Gold) standards across the program
• Establish ELT-first design principles using Azure Databricks and Delta Lake
• Design reusable, metadata-driven ETL frameworks supporting multiple ingestion patterns
• Define ingestion strategies for CDC, full loads, and streaming data from Azure Event Hub and databases
• Design and implement Databricks Auto Loader for scalable ingestion with schema drift handling
• Define merge and upsert strategies using Delta Lake for Silver and Gold layers
• Establish best practices for:
• Schema evolution and validation
• Late-arriving data handling
• Idempotent processing
• Define Delta Lake maintenance strategies (OPTIMIZE, VACUUM, Z-ORDER)
• Define partitioning strategies based on data volume, access patterns, and downstream usage
• Optimize Spark workloads for joins, aggregations, and large-scale transformations
• Ensure efficient cluster sizing and job configuration for cost and performance balance
• Define orchestration approaches using Azure Data Factory and Databricks Workflows
• Design dependency management across Bronze, Silver, and Gold pipelines
• Enable parameterized and reusable pipelines supporting multi-tenant and multi-source ingestion
• Define standardized error handling, retry, and recovery mechanisms
• Implement data quality checks and validation at each layer
• Design observability using Azure Monitor and Alerts
• Ensure pipeline resilience and operational stability
• Align ETL design with Azure security, governance, and lineage standards (Microsoft Purview)
• Design Gold-layer data models optimized for Synapse Dedicated SQL Pool, reporting, and analytics
• Support secure data sharing through Azure Data Share and external consumption platforms

Requirements:

• 8–10 years in data engineering and ETL/ELT development
• 4+ years designing and implementing cloud-based data platforms (Azure preferred)
• 2+ years in an architecture, lead, or technical design role
• Strong expertise in Azure Databricks architecture and Spark-based ETL
• Deep hands-on experience with Delta Lake (MERGE, schema evolution, ACID guarantees)
• Experience with Databricks Auto Loader for streaming and incremental ingestion
• Proven experience designing enterprise-grade ETL frameworks
• Strong knowledge of schema drift handling, CDC patterns, and incremental processing
• Hands-on experience with Azure Data Factory for orchestration
• Expertise in performance tuning and optimization for Databricks and Spark workloads
• Experience with real-time and streaming data pipelines
• Exposure to data migration and legacy system decommissioning programs
• Strong understanding of error handling, retry logic, and fault-tolerant pipeline design
• Strong experience with Azure ADLS Gen2, Event Hub, Databricks, Synapse
• Familiarity with Microsoft Purview or equivalent governance tools
• Experience supporting downstream analytics, reporting, and data sharing use cases
• Strong architectural thinking and decision-making ability
• Ability to define standards and mentor engineering teams
• Excellent communication and documentation skills
• Experience collaborating with platform, security, and analytics stakeholders
• Knowledge of CI/CD and DevOps practices for Databricks and data pipelines
• Experience working in large enterprise or multi-domain data programs