Design and develop data ingestion pipelines from external sources (ETLs, APIs, relational and non-relational databases, files, telemetry sensors, etc.);
Integrate and consolidate data from different sources within the Azure environment, ensuring data quality and consistency;
Implement reference architecture for data processing and storage;
Create and manage data lakes, data warehouses, and data marts within the Azure ecosystem;
Develop and optimize ETL/ELT processes using Microsoft tools such as Azure Data Factory, Databricks, Synapse Analytics, and Azure Functions;
Ensure governance, security, and scalability of data pipelines;
Monitor and optimize data load performance, identify bottlenecks, and propose improvements;
Work closely with data scientists and functional analysts to meet business needs.

Solid experience as a Data Engineer, focusing on building scalable pipelines;
Advanced knowledge of Azure Data Services, including Data Factory, Synapse, Databricks, Azure SQL, Blob Storage, and Monitoring;
Experience integrating data via REST/SOAP APIs and consuming real-time data streams;
Knowledge of relational databases (SQL Server, PostgreSQL) and non-relational databases (Cosmos DB, MongoDB);
Proficiency with Python, Spark, SQL, and PowerShell for data manipulation and processing;
Experience with DataOps, source control (Git), and CI/CD for data pipelines;
Knowledge of data modeling, query optimization, and database tuning;
Understanding of data security and compliance, including LGPD and best practices for storage and sharing;
Ability to document and communicate clearly for interaction with technical and business teams.

Differentials:

Experience with event-driven architectures using Azure Event Hub or Kafka;
Knowledge of machine learning pipelines and MLOps on Azure;
Microsoft certifications (DP-203 – Azure Data Engineer Associate or equivalent).

Senior Data Engineer – AI Automations, Azure, AWS

Key skills