Key skills
Azure Databricks architectureDelta LakeETL frameworks designAzure Data Factoryperformance tuningstreaming data pipelinesschema drift handlingarchitectural thinkingmentoring teamsSQLETLELTData EngineeringAnalyticsDatabricksSparkAzureAzure MonitorCI/CDCommunication
About this role
Algoworks is a global team of engineers and innovators focused on achieving client goals through technology. The ETL / ELT Architect will lead the design and governance of scalable data pipelines on Microsoft Azure, defining architecture standards and best practices for data ingestion and processing.
Responsibilities:
- Define and govern Medallion Architecture (Bronze, Silver, Gold) standards across the program
- Establish ELT-first design principles using Azure Databricks and Delta Lake
- Design reusable, metadata-driven ETL frameworks supporting multiple ingestion patterns
- Define ingestion strategies for CDC, full loads, and streaming data from Azure Event Hub and databases
- Design and implement Databricks Auto Loader for scalable ingestion with schema drift handling
- Define merge and upsert strategies using Delta Lake for Silver and Gold layers
- Establish best practices for: Schema evolution and validation, Late-arriving data handling, Idempotent processing
- Define Delta Lake maintenance strategies (OPTIMIZE, VACUUM, Z-ORDER)
- Define partitioning strategies based on data volume, access patterns, and downstream usage
- Optimize Spark workloads for joins, aggregations, and large-scale transformations
- Ensure efficient cluster sizing and job configuration for cost and performance balance
- Define orchestration approaches using Azure Data Factory and Databricks Workflows
- Design dependency management across Bronze, Silver, and Gold pipelines
- Enable parameterized and reusable pipelines supporting multi-tenant and multi-source ingestion
- Define standardized error handling, retry, and recovery mechanisms
- Implement data quality checks and validation at each layer
- Design observability using Azure Monitor and Alerts
- Ensure pipeline resilience and operational stability
- Align ETL design with Azure security, governance, and lineage standards (Microsoft Purview)
- Design Gold-layer data models optimized for Synapse Dedicated SQL Pool, reporting, and analytics
- Support secure data sharing through Azure Data Share and external consumption platforms
Requirements:
- 8–10 years in data engineering and ETL/ELT development
- 4+ years designing and implementing cloud-based data platforms (Azure preferred)
- 2+ years in an architecture, lead, or technical design role
- Strong expertise in Azure Databricks architecture and Spark-based ETL
- Deep hands-on experience with Delta Lake (MERGE, schema evolution, ACID guarantees)
- Experience with Databricks Auto Loader for streaming and incremental ingestion
- Proven experience designing enterprise-grade ETL frameworks
- Strong knowledge of schema drift handling, CDC patterns, and incremental processing
- Hands-on experience with Azure Data Factory for orchestration
- Expertise in performance tuning and optimization for Databricks and Spark workloads
- Experience with real-time and streaming data pipelines
- Exposure to data migration and legacy system decommissioning programs
- Strong understanding of error handling, retry logic, and fault-tolerant pipeline design
- Strong experience with Azure ADLS Gen2, Event Hub, Databricks, Synapse
- Familiarity with Microsoft Purview or equivalent governance tools
- Experience supporting downstream analytics, reporting, and data sharing use cases
- Strong architectural thinking and decision-making ability
- Ability to define standards and mentor engineering teams
- Excellent communication and documentation skills
- Experience collaborating with platform, security, and analytics stakeholders
- Knowledge of CI/CD and DevOps practices for Databricks and data pipelines
- Experience working in large enterprise or multi-domain data programs