AWS Data Engineer – Open Data Platform

The client is seeking an experienced AWS Data Engineer to support an open data platform. The role focuses on designing cloud-native data pipelines, architecting Apache Iceberg tables, integrating AWS and Snowflake environments, and optimizing data processing performance, security, and cost.

Responsibilities:

• Design and develop end-to-end data ingestion, transformation, and processing pipelines using AWS Glue, Apache Spark, and Amazon S3
• Build and manage Apache Iceberg tables to enable open, interoperable access across multiple compute engines
• Configure and integrate Apache Polaris with Iceberg REST Catalog for AWS and Snowflake environments
• Implement table maintenance, schema evolution, and partitioning strategies to optimize performance and cost
• Support platform security, access controls, and AWS networking infrastructure
• Conduct workload benchmarking and performance analysis comparing AWS Glue versus Snowflake for cost and efficiency
• Provide operational monitoring, troubleshooting, and optimization of cloud-based data platform infrastructure

Requirements:

• 8–10 years of experience in data engineering and cloud-based data pipeline development
• Proven expertise with Amazon S3, AWS Glue, Apache Spark, and large-scale data processing
• Strong hands-on experience with Apache Iceberg architecture, including table design and maintenance
• Demonstrated proficiency with Apache Iceberg schema evolution and partitioning optimization
• Experience configuring and deploying Apache Polaris and Iceberg REST Catalog
• Solid understanding of AWS IAM, networking, and security best practices in cloud environments
• Experience with performance tuning, cost optimization, and cross-platform data tool evaluation