AWS Data Engineer w/d Collibra

ApTask is seeking an AWS Data Engineer with experience in Collibra and Atacama. The role involves analyzing and migrating legacy data processes to AWS, developing data pipelines, and ensuring data quality and integration between systems.

Responsibilities:

• Analyze legacy batch jobs, scripts, database procedures, and data workflows to understand current-state processing, dependencies, business logic, scheduling requirements, and operational constraints
• Rewrite and migrate legacy batch workloads into AWS Batch using modern engineering practices while preserving business functionality and meeting existing service-level agreements
• Develop, enhance, and maintain data pipelines and batch processing solutions using Java Spring Batch, Python, SQL, shell scripting, and AWS services
• Support integration between on-premises legacy systems and AWS-based target platforms, ensuring reliable data movement, synchronization, monitoring, and operational continuity
• Build and optimize data ingestion workflows to support the Rapid Data Ingestion platform, including file intake, validation, transformation, inventory tracking, and downstream consumption
• Implement data quality checks to detect changes in vendor file formats, data integrity issues, missing or incomplete data, and other anomalies that may impact critical business processes
• Enable and support data lineage capabilities by capturing source-to-target data movement, transformations, metadata, and audit-relevant processing details
• Support disaster recovery implementation for the Rapid Data Ingestion platform in AWS to improve resiliency and continuity for critical and audit-sensitive processes

Requirements:

• Strong hands-on experience in Collibra / Atacama
• Analyze legacy batch jobs, scripts, database procedures, and data workflows to understand current-state processing, dependencies, business logic, scheduling requirements, and operational constraints
• Rewrite and migrate legacy batch workloads into AWS Batch using modern engineering practices while preserving business functionality and meeting existing service-level agreements
• Develop, enhance, and maintain data pipelines and batch processing solutions using Java Spring Batch, Python, SQL, shell scripting, and AWS services
• Support integration between on-premises legacy systems and AWS-based target platforms, ensuring reliable data movement, synchronization, monitoring, and operational continuity
• Build and optimize data ingestion workflows to support the Rapid Data Ingestion platform, including file intake, validation, transformation, inventory tracking, and downstream consumption
• Implement data quality checks to detect changes in vendor file formats, data integrity issues, missing or incomplete data, and other anomalies that may impact critical business processes
• Enable and support data lineage capabilities by capturing source-to-target data movement, transformations, metadata, and audit-relevant processing details
• Support disaster recovery implementation for the Rapid Data Ingestion platform in AWS to improve resiliency and continuity for critical and audit-sensitive processes