Key skills
AirflowAmazon RedshiftAWSAzureCloudPostgresPySparkPythonSparkSQLTerraformMachine LearningSnowflakeRedshiftGitHub ActionsCloudFormationGluePostgreSQLGitHubCI/CD
About this role
Role Overview
- Lead the design and evolution of data pipelines in cloud environments (AWS);
- Drive the migration of data solutions from Azure to AWS;
- Define and implement modern architectures (Lakehouse, Data Lakes);
- Ensure data efficiency, scalability and governance (catalog, lineage, compliance);
- Orchestrate robust, production-grade pipelines;
- Set up environments with CI/CD and engineering best practices;
- Support data initiatives for Machine Learning;
- Serve as a technical reference, proposing improvements and strategic direction.
Requirements
- Proven experience as a Senior Data Engineer;
- Strong proficiency in Python, Spark (PySpark) and SQL;
- Experience with orchestration tools (Airflow, Step Functions, or ADF);
- Familiarity with Data Lakes and Lakehouse architectures (Delta, Iceberg, Parquet);
- Experience with AWS services (EMR, Glue, Lake Formation);
- Knowledge of Azure (Data Factory, Synapse, etc.);
- Experience with databases such as Redshift, Snowflake, PostgreSQL;
- Practice with CI/CD (Terraform, GitHub Actions, CloudFormation);
- Nice to have: experience migrating between clouds (Azure → AWS);
- Experience working with data for Machine Learning;
- Experience in high-scale environments.
Tech Stack
- Airflow
- Amazon Redshift
- AWS
- Azure
- Cloud
- Postgres
- PySpark
- Python
- Spark
- SQL
- Terraform
Benefits
- Health insurance
- Private pension plan
- Wellness program
- Continuous learning