Senior Data Platform Engineer

NCC Group is a global cybersecurity company that assesses, develops, and manages cyber threats. They are seeking a Senior Data Platform Engineer to design and build scalable data pipelines that support advanced analytics and machine learning solutions.

Responsibilities:

• Designing, building, and optimising scalable data pipelines that power advanced analytics and machine learning solutions
• Develop robust data pipelines that feed analytics and machine learning tools such as Amazon SageMaker and third-party platforms like Databricks
• Leverage AWS technologies such as EMR, S3, EKS and Airflow to process and orchestrate high-volume datasets
• Ensure solutions are scalable, resilient and cost-efficient
• Embed data loss prevention (DLP) principles and controls into data pipelines to protect sensitive information
• Ensure data is reliable, accessible, well-governed and optimised for downstream consumption

Requirements:

• Strong experience in data engineering within AWS cloud environments
• Hands-on experience with AWS big data technologies such as EMR, S3 and SageMaker
• Proficiency in Python for building scalable data pipelines and processing frameworks. Experience with Apache Spark for distributed data processing
• Experience designing and maintaining scalable batch and real-time data pipelines
• Solid understanding of ETL/ELT design patterns and data modelling techniques
• Experience with workflow orchestration tools such as Apache Airflow (ideally deployed on AWS)
• Familiarity with containerisation and orchestration using Docker and Kubernetes (EKS)
• Experience with infrastructure as code (e.g. Terraform) and CI/CD/GitOps practices
• Proven ability to optimise performance and reduce cloud costs through partitioning, clustering and workload management
• Understanding of data security principles, including data loss prevention (DLP)
• Experience with Databricks or similar third-party big data platforms
• Knowledge of real-time streaming technologies (e.g. Kafka, Kinesis)
• Experience implementing data governance and compliance frameworks
• Familiarity with monitoring and observability tools in AWS environments
• Exposure to Lakehouse or modern data platform architectures