Azure Data Engineer Lead

Tata Consultancy Services is seeking an Azure Data Engineer Lead to work across various Fabric workloads and design data architectures for enterprise-scale analytics. The role involves preparing and enriching data for analysis, ensuring data quality, and collaborating with stakeholders to translate requirements into scalable solutions.

Responsibilities:

• Work across Fabric workloads: COSMSO, Kusto, Data Factory (ETL), Synapse Data Engineering (Spark/SQL), Synapse Data Warehousing (SQL), and OneLake for storage
• Design data loading patterns, Lakehouse architectures, and orchestration processes for enterprise-scale analytics
• Prepare and enrich data for analysis, ensuring data quality and consistency checks
• Implement semantic models for BI and self-service analytics
• Secure and manage analytics assets, enforce governance policies, and monitor compliance and Managed Identity
• Monitor and optimize Data pipelines and workloads for cost efficiency and high availability
• Work closely with architects, analysts, and business teams to translate requirements into scalable solutions
• Enable Power BI integration for visualization and reporting
• Build and manage data pipelines using COSMOS, Azure Data Factory (ADF), Synapse, Fabric pipelines for batch and incremental ingestion
• Automate ETL workflows and implement parameterized pipelines for scalability
• Develop ETL code / pipelines to clean, transform, and convert raw files into optimized formats (Parquet/Delta)
• Create views, stored procedures, and dimensional models for reporting and analytics
• Monitor Dynamic Management Views (DMVs) in Synapse to analyze system health and reduce bottlenecks
• Implement workload management strategies for query performance and resource governance
• Apply role-based authentication, encryption/decryption logic, and compliance controls
• Automate governance tasks to reduce manual intervention
• Build failure monitoring systems integrated with Logic Apps for alerting and proactive issue resolution

Requirements:

• Synapse
• SQL Server
• Python
• Azure Storage
• ADF
• Cosmos
• Kusto
• Digital: PySpark
• Fabric
• Data Engineering
• DataWarehouse
• Work across Fabric workloads: COSMSO, Kusto, Data Factory (ETL), Synapse Data Engineering (Spark/SQL), Synapse Data Warehousing (SQL), and OneLake for storage
• Design data loading patterns, Lakehouse architectures, and orchestration processes for enterprise-scale analytics
• Prepare and enrich data for analysis, ensuring data quality and consistency checks
• Implement semantic models for BI and self-service analytics
• Secure and manage analytics assets, enforce governance policies, and monitor compliance and Managed Identity
• Monitor and optimize Data pipelines and workloads for cost efficiency and high availability
• Work closely with architects, analysts, and business teams to translate requirements into scalable solutions
• Enable Power BI integration for visualization and reporting
• Build and manage data pipelines using COSMOS, Azure Data Factory (ADF), Synapse, Fabric pipelines for batch and incremental ingestion
• Automate ETL workflows and implement parameterized pipelines for scalability
• Develop ETL code / pipelines to clean, transform, and convert raw files into optimized formats (Parquet/Delta)
• Create views, stored procedures, and dimensional models for reporting and analytics
• Monitor Dynamic Management Views (DMVs) in Synapse to analyze system health and reduce bottlenecks
• Implement workload management strategies for query performance and resource governance
• Apply role-based authentication, encryption/decryption logic, and compliance controls
• Automate governance tasks to reduce manual intervention
• Build failure monitoring systems integrated with Logic Apps for alerting and proactive issue resolution
• Strong in Azure Data Services ADF, Azure SQL database, Synapse
• Ability to understand vast amounts of data, identify and fix data issues
• Strong in database and DW concepts
• Excellent technical & analytical skills with strong business acumen
• Strong communication skills
• Must have Microsoft Azure Cloud and services experience
• Generic Managerial Skills, If any
• Azure OpenAI