Design and develop data ingestion pipelines from external sources (ETLs, APIs, relational and non-relational databases, files, telemetry sensors, etc.);
Integrate and consolidate data from different sources within the Azure environment, ensuring data quality and consistency;
Implement reference architecture for data processing and storage;
Create and manage data lakes, data warehouses, and data marts within the Azure ecosystem;
Develop and optimize ETL/ELT processes using Microsoft tools such as Azure Data Factory, Databricks, Synapse Analytics, and Azure Functions;
Ensure governance, security, and scalability of data pipelines;
Monitor and optimize data load performance, identify bottlenecks, and propose improvements;
Work closely with data scientists and functional analysts to meet business needs.
Requirements
Solid experience as a Data Engineer, focusing on building scalable pipelines;
Advanced knowledge of Azure Data Services, including Data Factory, Synapse, Databricks, Azure SQL, Blob Storage, and Monitoring;
Experience integrating data via REST/SOAP APIs and consuming real-time data streams;
Knowledge of relational databases (SQL Server, PostgreSQL) and non-relational databases (Cosmos DB, MongoDB);
Proficiency with Python, Spark, SQL, and PowerShell for data manipulation and processing;
Experience with DataOps, source control (Git), and CI/CD for data pipelines;
Knowledge of data modeling, query optimization, and database tuning;
Understanding of data security and compliance, including LGPD and best practices for storage and sharing;
Ability to document and communicate clearly for interaction with technical and business teams.
Differentials:
Experience with event-driven architectures using Azure Event Hub or Kafka;
Knowledge of machine learning pipelines and MLOps on Azure;
Microsoft certifications (DP-203 – Azure Data Engineer Associate or equivalent).
Tech Stack
Azure
ETL
Kafka
MongoDB
Postgres
Python
SOAP
Spark
SQL
Benefits
Work mode: Remote (Home Office);
Possible travel to client sites (Campinas/SP and others), paid by the company;
Contract type: PJ (contractor/B2B);
Team working hours: Business hours / to be specified.