Plan and execute the migration of workloads, notebooks, and data between Databricks workspaces;
Refactor and optimize Databricks Notebooks (Python/PySpark/SQL) to ensure compatibility and performance in the new environment;
Implement Delta Lake architecture with a focus on cost optimization and performance on AWS;
Implement Unity Catalog for centralized metadata management, data lineage, and access control (ACLs);
Define governance policies, ensuring compliance with LGPD/GDPR and row
and column-level security;
Ensure data quality and integrity during and after the migration process;
Build data ingestion pipelines originating from microservices-based architectures;
Manage the extraction and transformation of data from NoSQL databases, specifically MongoDB, into the Data Lake;
Orchestrate data flows using native AWS tools or Databricks Workflows;
Provide optimized, gold-standard datasets for consumption in Tableau;
Collaborate with business analysts to ensure data modeling meets the performance requirements of Tableau dashboards.

Proven experience in development, cluster administration, and extensive use of Databricks Notebooks;
Hands-on experience with Data Lake migrations or moving workloads between environments (e.g., Legacy Hive Metastore to Unity Catalog);
Strong expertise in implementing Unity Catalog, data lineage, and permission management;
AWS ecosystem: solid experience with S3, IAM, EC2, and data-related networking services;
NoSQL & Microservices: practical experience handling JSON documents and extracting data from MongoDB;
Ability to integrate Databricks with Tableau (connectors, query optimization, publishing data sources);
Proficiency in Python (PySpark);
Advanced SQL skills.

Senior AWS Data Engineer

Key skills