Key skills
AzureCloudETLHDFSPySparkPythonSparkSQLELTData LakeDatabricksCI/CDChange ManagementCommunicationCollaboration
About this role
Role Overview
- Design, develop and maintain scalable, robust data pipelines.
- Build data ingestion, transformation and modeling solutions using Databricks, Spark/PySpark, Cloudera and Azure Data Factory (ADF).
- Ensure data quality, integrity and usability across the entire pipeline.
- Implement CI/CD pipelines focused on ETL processes.
- Design and maintain Data Lake and Data Warehouse solutions, applying data governance best practices.
- Apply FinOps concepts to optimize cloud data processing costs.
- Develop and document data workflows, processes and architectures.
- Perform technical troubleshooting of data pipelines and data infrastructure.
- Work with concepts and structures such as Bronze, Silver and Gold layers, Star Schema, Delta Tables, Delta Sharing and analytical tables.
- Collaborate with technical and business teams to orchestrate solutions aligned with company objectives.
- Manage the technical roadmap for data projects, considering dependencies, trade-offs and change management.
Requirements
- Bachelor's degree in Computer Science, Engineering, Information Technology or related fields.
- Proven experience as a Data Engineer or in a similar role.
- Advanced knowledge of:
- SQL
- Python / PySpark
- Azure Databricks (workflows, jobs, delta tables, queries, etc.)
- Azure Data Factory (ADF)
- Cloudera (Hive, Impala, HDFS, etc.)
- ETL/ELT and data pipelines
- Dimensional data modeling
- Data Lake / Data Warehouse
- Data governance and data quality
- Experience with CI/CD practices applied to data environments.
- Strong communication and collaboration skills with multidisciplinary teams.
- Ability to operate in dynamic environments and handle loosely defined technical scope.
- Knowledge of code optimization in cloud environments.
- Differential: Experience with Teradata.
- Experience with SAS.
Tech Stack
- Azure
- Cloud
- ETL
- HDFS
- PySpark
- Python
- Spark
- SQL
Benefits
- Health and dental insurance
- Meal and food allowance
- Childcare assistance
- Extended parental leave
- Partnerships with gyms and health and wellness professionals via Wellhub (Gympass) TotalPass
- Profit Sharing (PLR)
- Life insurance
- Continuous learning platform (CI&T University)
- Discount club
- Free online platform dedicated to promoting physical and mental health and well-being
- Pregnancy and responsible parenting course
- Partnerships with online course platforms
- Language learning platform
- And many others.