Key skills
AirflowApacheAWSAzureCloudDockerGrafanaKafkaKubernetesPythonSparkSQLTerraformRShellBashAIData LakeAnalyticsDatabricksApache AirflowdbtEKSAKSCloudWatchVersion ControlCI/CDLeadershipMentoring
About this role
Role Overview
- Modify or create cloud resources (AWS and Azure)
- Modify or create new resources in Databricks
- Modify or create new application resources in Kubernetes
- Propose and implement architectural improvements
- Implement LGPD-related controls in the data lake
- Implement Databricks metadata dashboards and analyses for control and compliance
- Conduct audits together with the governance team to align the data lake with the company's security policies
- Monitor and analyze costs to identify bottlenecks and cost-reduction opportunities
- Collaborate with data analytics and AI teams to optimize jobs and resource usage
- Work with other platform engineers to support skill development
- Serve as a technical point of reference for platform-related questions for data, business, and product stakeholders
Requirements
- Bachelor's degree or higher
- Solid knowledge of relational and non-relational databases: patterns, modeling and use cases
- Experience with programming languages for maintaining data pipelines (Python, shell/bash) and file manipulation for this purpose (e.g., JSON, Parquet, CSV)
- Experience consuming APIs and performing data cleaning, processing, deduplication, enrichment and normalization
- Experience with messaging infrastructure using Kafka + Debezium
- Experience with Kubernetes cluster infrastructure such as EKS and AKS
- Strong SQL skills and basic knowledge of DBT
- Experience with cloud computing projects (primarily AWS and/or Azure)
- Experience with Gitflow and version control
- Experience with Infrastructure as Code (Terraform)
- Experience with distributed processing (Databricks/Spark)
- Experience with Docker
- Experience with orchestration tools such as Apache Airflow and Databricks Workflows
- Knowledge of data architecture
- Experience building CI/CD pipelines
- Experience with observability tools (Grafana / CloudWatch)
Tech Stack
- Airflow
- Apache
- AWS
- Azure
- Cloud
- Docker
- Grafana
- Kafka
- Kubernetes
- Python
- Spark
- SQL
- Terraform
Benefits
- Meal allowance / meal card
- Flexible working hours and remote arrangements (for remote positions)
- Commuter allowance / transportation voucher (for hybrid or on-site positions)
- Profit sharing (PLR)
- Flexible benefits via Flash Card (Multibenefits)
- Gympass / Wellhub
- Psicologia Viva (online platform for therapy and nutrition consultations)
- Health and dental insurance
- Life insurance
- Extended parental leave (up to 6 months for mothers and 20 days for fathers)
- Rede D'Ór: support and important information for maternal and newborn health with a network of accredited nurses
- Partnership with your regional SESC (varied programs in education, health, culture, leisure and social assistance)
- Birthday Day Off (one paid day off on or during your birthday month)
- Online learning platform with a variety of courses to enhance your skills (UCA)
- Language academy (AIA)
- Leadership development program
- Mentoring for Afya Women (MMA)
- Discounts on undergraduate and graduate courses at Afya's educational institutions