Key skills
AzureCloudETLPythonSQLAIMLRAGPineconeELTData EngineeringAnalyticsSnowflake
About this role
Role Overview
- Design and build ingestion pipelines: Azure source systems → ADF → Snowflake (batch and near-real-time)
- Build document ingestion pipelines: File Servers → chunking → Pinecone (RAG layer)
- Implement data transformation and cleansing in ADF; ensure Snowflake data is analytics-ready
- Manage Snowflake schema design and performance optimisation
- Collaborate with AI/ML engineers on data quality.
- Implement Data access controls
Requirements
- 5–8 years data engineering with strong focus on cloud-native pipelines
- Azure Data Factory, pipeline design and development
- Strong SQL and data modelling; Snowflake required
- ETL/ELT patterns and transformation tools
- Python scripting for data processing, file parsing (PDFs, Office docs)
- Document ingestion pipelines feeding vector databases or search indexes
- Data governance: metadata management, lineage, access controls, audit logging
Tech Stack
- Azure
- Cloud
- ETL
- Python
- SQL
Benefits
- Opportunities for professional growth and development.
- A collaborative and inclusive work environment.
- The chance to work on impactful projects with a talented team.
- Excellent compensation in USD.
- Hardware and software setup.