Develop and maintain robust, scalable, and testable data pipelines using PySpark and Airflow.
Design and implement data ingestion and transformation processes to populate the data lake using a layered architecture (Bronze, Silver, Gold).
Work on data quality control, documentation, and lineage management using OpenMetadata.
Collaborate with product and capture squads to ensure data consistency and coverage.
Ensure data governance, versioning, and auditing of production pipelines.
Optimize ETL/ELT routines and query performance in relational databases, data warehouses, and engines such as Elasticsearch and Athena.
Requirements
We are looking for people with knowledge in...
PySpark
Apache Airflow
AWS S3, Glue, Athena, EC2
SQL (Athena, PostgreSQL)
Elasticsearch/OpenSearch
Docker
Pandas
Jupyter
Unix (Linux), Bash
DBT
Nice to have:
Glue
Delta Lake
Kubernetes
NoSQL
Elasticsearch
Airbyte
Tech Stack
Airflow
Apache
AWS
Docker
EC2
ElasticSearch
ETL
Kubernetes
Linux
NoSQL
Pandas
Postgres
PySpark
SQL
Unix
Benefits
Swile benefits card with a fixed monthly value of R$2,540.00 (food, mobility, multi-balance, and home office allowance);
National health plan (Unimed or Amil);
Dental plan (Odontoprev);
Life insurance (MetLife);
TotalPass;
Starbem (health services platform for your physical, mental, and emotional well-being);
Pharmacy discount program with Panvel;
Extended maternity and paternity leave through the Empresa Cidadã program;
Subsidy for professional development in partnership with Unico Skill, offering various options for undergraduate, postgraduate, language courses, mentorships, etc.;
Private English lessons for leadership and specialists level II and above;
School/Education assistance;
Fresh fruit, cookies, coffee, tea, and energy drinks available at any time;
Celebrations, integration events, and team building activities;
Partnership with KÜK Station to provide the best for our Loggers on office days;
In-company massage;
Birthday day-off;
Birthday gift;
Service anniversary gifts;
Recruta Loggers (employee referral program with bonuses).