Work with Technical architects, Product Owners, and Business teams to translate requirements into technical design for data modelling and data integration
Demonstrate deep background in data warehousing, data modelling and ETL/ELT data processing patterns
Design and develop ETL/ELT pipelines with reusable patterns and frameworks
Design and build efficient SQLs to process and curate the data sets in HANA, Azure, and Snowflake
Design and review data ingestion frameworks leveraging Python, Spark, Azure Data Factory, Snowpipe, etc.
Design and build Data Quality models and ABCR frameworks to ingest, validate, curate, and prepare the data for consumption
Understand the functional domain, business needs and able to identify the gaps in the requirements proactively prior to implementing solutions
Work with platform teams to design and build processes for automation in pipeline build, testing and code migrations
Collaborate with Data Scientists to build and maintain scalable pipelines in Azure Data Factory and Databricks that support AI/ML modelling and model training workflows
Support the integration of Generative AI by building robust pipelines for high-quality training data and implementing vector databases to power Retrieval-Augmented Generation (RAG) workflows
Demonstrate exceptional impact in delivering projects, products and/or platforms in terms of scalable data processing and application architectures, technical deliverables, and delivery throughout the project lifecycle
Provide design and guiding principles on building data models and semantic models in Snowflake – enabling true self-service
Responsible for ensuring the effectiveness of the ingestion and data delivery frameworks and patterns
Build and maintain data development standards and principles, provide guidance and project specific recommendations as appropriate
Must be conversant with DevOps delivery approach and tools and have a track record of delivering products in agile model
Provide insight and direction on roles and responsibilities required for platform/ product operations
Requirements
5+ years of experience designing, developing, and building ETL/ELT pipelines, procedures, and SQLs on MPP platforms such as HANA and Snowflake
5+ years of experience in Data warehousing and Business Intelligence
Experience in Data Warehousing concepts like Star schema, Snowflake schema, Fact table and Dimension Table
Experience in using various features of ADF to load data into snowflake
Experience in designing and building metadata driven data ingestion frameworks, building Azure Data Factory, SnowSQL, Snowpipe
Hands-on experience with Azure
Familiarity in leveraging Azure Stream Analytics, Azure Analysis Services, Data Lake Analytics, HDInsight, HDP, Spark, Databricks, MapReduce, Pig, Hive, Tez, SSAS, Watson Analytics, SPSS
Strong Knowledge on source code management, configuration management, CI/CD, security, and performance
Ability to look ahead to identify opportunities and thrive in a culture of innovation
Self-starter who can see the big picture, and prioritize your work to make the largest impact on the business’ and customer’s vision and requirements
Experience in building, testing, and deploying code to run on Azure cloud data lake
Ability to Lead/nurture/mentor others in the team
A can-do attitude in anticipating and resolving problems to help your team to achieve its goals
Must have experience in Agile development methods
Tech Stack
Azure
Cloud
ETL
MapReduce
Python
Spark
Benefits
Innovation, growth, and the chance to make a real impact
Environment that promotes sustainability, inclusion, wellbeing, and career development