Sr. Data Engineer

Shell is a leading company in the energy sector, and they are seeking a highly skilled Sr. Data Engineer to join their team. The role involves designing, building, and maintaining data infrastructure to support optimal data extraction, transformation, and loading from various sources, while collaborating with multiple business stakeholders to ensure business value generation.

Responsibilities:

• Closely working with product / project team and stakeholders to understand the requirement
• Solution consultation and development in accordance with the enterprise technology principles, methods, standards, and practices
• Estimation, solution & detailed technical design, implementation of high performance data pipelines which is secure and scalable
• To define data models and schemas to support data analytics
• Integrate different kind of data sources(database, API, streams) into unified data platform
• Implement solutions for near-real-time data access
• Maintain and improve data quality, consistency and data governance throughout the ETL/ELT process
• Application technical deliverables including code, associated tests, and documentation
• To automate the data workflows and reduce manual intervention
• To collaborate with multiple teams to integrate data solutions into other applications
• To build the reusable solutions which can be reused across multiple projects
• To identify areas, for improving efficiency in data operations
• To innovate solutions for complex data and to ensure scalability and performance

Requirements:

• Must have legal authorization to work in the US on a full-time basis for anyone other than current employer
• Degree in Software Engineer or demonstrated capability, applied experience, and role-relevant skills
• 10 + Years of relevant data engineering experience which include solutioning, design, planning and implementing end-to-end data solutions. Have in-depth hands-on experience in multiple complex implementations which include different tools and technologies
• Good understanding of Energy Trading Business Domain and experience in transformation, orchestration of data in Energy Trading area
• Proficient in Azure Cloud Technologies: Hands-on experience with a wide range of Azure services including Azure Data Factory (ADF), Azure Databricks (ADB), Azure Active Directory, Azure Storage, Azure Data Lake Services (ADLS), Azure Key Vault, Azure SQL Database, Azure Logic Apps and Azure DevOps
• Data Engineering Expertise: Implemented several end-to-end data solutions using Azure Databricks Delta Lake for optimized storage, processing, and analytics over Azure Data Lake Storage (ADLS)
• Azure Data Integration & Orchestration: Extensive experience in using Azure Data Factory (ADF) to orchestrate complex ETL/ELT pipelines, integrating with various data sources and destinations across Azure ecosystem
• Advanced Knowledge in Data Processing & Analytics: Strong skills in Apache Spark and PySpark, leveraging Databricks for large-scale data processing and transformation
• Azure Functions & Event-Driven Architectures: Experience developing serverless solutions using Azure Functions for data integration, real-time message consumption, and event-driven architectures
• Python Programming: Expertise in developing data engineering pipelines, automation scripts, and data transformations using Python
• Data Storage Solutions: Hands-on experience with Azure SQL Database, SQL Warehouse, and Blob Storage/ADLS for structured, semi-structured data storage
• DevOps, GitHub & CI/CD Pipelines: Skilled in Azure DevOps/GitHub for implementing continuous integration and continuous deployment (CI/CD) pipelines and managing source control and project workflows