Key skills
AzureCloudETLKafkaPySparkPythonSQLAIMachine LearningMLMLOpsELTData EngineeringDatabricksGitHubVersion ControlCI/CD
About this role
Role Overview
- Develop and maintain data pipelines (ETL/ELT) on Azure, using Databricks for processing and integration.
- Maintain a 360° perspective (e.g., understand the project as a whole and the impacts that point solutions may have on the overall project).
- Implement CI/CD practices to automate deployments and manage code versioning in GitHub.
- Design and optimize data models to ensure scalability, performance, and governance.
- Support cloud migration projects, ensuring data security, quality, and integrity.
- Collaborate with business and data science teams to deploy AI models in cloud environments.
- Ensure data quality and consistency by applying data profiling and monitoring techniques.
- Document processes, architectures, and data engineering best practices.
Requirements
- Strong experience with Azure Data Services and Databricks.
- Experience with Databricks using PySpark.
- Productionizing Machine Learning models on Databricks (MLOps).
- Near-real-time data processing (e.g., Kafka).
- Proficient in Python and SQL for data manipulation and transformation.
- Knowledge of CI/CD and version control with GitHub.
- Experience in data modeling (conceptual, logical, and physical).
- Experience with cloud migration projects.
- Experience with ETL/ELT and integrating large volumes of data.
- Experience deploying AI models and using tools such as Azure ML.
- Experience with AI projects and automating pipelines.
Tech Stack
- Azure
- Cloud
- ETL
- Kafka
- PySpark
- Python
- SQL
Benefits
- Health and dental insurance;
- Meal and food allowances;
- Childcare allowance;
- Extended parental leave;
- Partnerships with gyms and health & wellness professionals via Wellhub (Gympass) / TotalPass;
- Profit Sharing (PLR);
- Life insurance;
- Continuous learning platform (CI&T University);
- Employee discount club;
- Free online platform dedicated to physical and mental health and well-being;
- Pregnancy and responsible parenthood course;
- Partnerships with online course platforms;
- Language learning platform;
- And many more.