Senior Data Engineer

Role Overview

• Work with Technical architects, Product Owners, and Business teams to translate requirements into technical design for data modelling and data integration
• Demonstrate deep background in data warehousing, data modelling and ETL/ELT data processing patterns
• Design and develop ETL/ELT pipelines with reusable patterns and frameworks
• Design and build efficient SQLs to process and curate the data sets in HANA, Azure, and Snowflake
• Design and review data ingestion frameworks leveraging Python, Spark, Azure Data Factory, Snowpipe, etc.
• Design and build Data Quality models and ABCR frameworks to ingest, validate, curate, and prepare the data for consumption
• Understand the functional domain, business needs and able to identify the gaps in the requirements proactively prior to implementing solutions
• Work with platform teams to design and build processes for automation in pipeline build, testing and code migrations
• Collaborate with Data Scientists to build and maintain scalable pipelines in Azure Data Factory and Databricks that support AI/ML modelling and model training workflows
• Support the integration of Generative AI by building robust pipelines for high-quality training data and implementing vector databases to power Retrieval-Augmented Generation (RAG) workflows
• Demonstrate exceptional impact in delivering projects, products and/or platforms in terms of scalable data processing and application architectures, technical deliverables, and delivery throughout the project lifecycle
• Provide design and guiding principles on building data models and semantic models in Snowflake – enabling true self-service
• Responsible for ensuring the effectiveness of the ingestion and data delivery frameworks and patterns
• Build and maintain data development standards and principles, provide guidance and project specific recommendations as appropriate
• Must be conversant with DevOps delivery approach and tools and have a track record of delivering products in agile model
• Provide insight and direction on roles and responsibilities required for platform/ product operations

Requirements

• 5+ years of experience designing, developing, and building ETL/ELT pipelines, procedures, and SQLs on MPP platforms such as HANA and Snowflake
• 5+ years of experience in Data warehousing and Business Intelligence
• Experience in Data Warehousing concepts like Star schema, Snowflake schema, Fact table and Dimension Table
• Experience in using various features of ADF to load data into snowflake
• Experience in designing and building metadata driven data ingestion frameworks, building Azure Data Factory, SnowSQL, Snowpipe
• Hands-on experience with Azure
• Familiarity in leveraging Azure Stream Analytics, Azure Analysis Services, Data Lake Analytics, HDInsight, HDP, Spark, Databricks, MapReduce, Pig, Hive, Tez, SSAS, Watson Analytics, SPSS
• Strong Knowledge on source code management, configuration management, CI/CD, security, and performance
• Ability to look ahead to identify opportunities and thrive in a culture of innovation
• Self-starter who can see the big picture, and prioritize your work to make the largest impact on the business’ and customer’s vision and requirements
• Experience in building, testing, and deploying code to run on Azure cloud data lake
• Ability to Lead/nurture/mentor others in the team
• A can-do attitude in anticipating and resolving problems to help your team to achieve its goals
• Must have experience in Agile development methods

Tech Stack

• Azure
• Cloud
• ETL
• MapReduce
• Python
• Spark

Benefits

• Innovation, growth, and the chance to make a real impact
• Environment that promotes sustainability, inclusion, wellbeing, and career development