Data Engineering Lead

MediaRadar is an industry leader in marketing intelligence, now including the data and capabilities of Vivvix. They are seeking a visionary and hands-on Data Engineering Lead to spearhead the design, development, and optimization of their next-generation data platform while ensuring alignment with business goals and collaborating with cross-functional teams.

Responsibilities:

• Design and supervise the implementation of comprehensive data pipelines utilizing Azure Databricks and PySpark
• Direct a team of data engineers, performing code reviews, offering technical expertise, and cultivating a culture of ongoing learning
• Develop high-performance schemas in PostgreSQL and refine complex SQL queries for large datasets
• Establish and apply optimal practices for data ingestion, transformation, and storage (Delta Lake/Lakehouse patterns)
• Collaborate closely with Data Analysts, Architects, and Product Managers to convert business requirements into technical specifications
• Promote the implementation of CI/CD, unit testing, and automated monitoring to achieve 99.9% data reliability
• Ensure data quality, governance, and compliance through validation, documentation, and secure practices
• Continuously improve data systems for enhanced performance, reliability, and scalability
• Effectively engage within an agile, cross-functional project team

Requirements:

• Expert-level experience managing workspaces, clusters, and job scheduling in Azure Databricks
• Solid understanding of data lakehouse architectures and Delta Lake
• Proven experience in Performance Tuning, Spark Optimization and Cost Reduction
• Advanced proficiency in PySpark, including Spark DataFrame APIs and Spark SQL for large-scale data processing involving various data formats
• Exceptional ability to write, tune, and troubleshoot complex SQL queries
• Hands-on experience with PostgreSQL, including relational database design, indexing, and performance optimization
• Proven track record of building scalable ETL/ELT data pipelines from scratch
• 10+ years of experience in Data Engineering or Software Engineering
• 3+ years as a formal technical Lead managing an agile team and implementing E2E solutions
• Bachelor's or Master's degree in Computer Science, Engineering, or a related field
• Strong communication skills with the ability to explain complex technical concepts to non-technical stakeholders
• Strong problem-solving skills and attention to detail
• Experience with Apache Airflow for managing complex task dependencies
• Familiarity with Azure Kubernetes Service (AKS) for deploying containerized data services
• Knowledge of Terraform or Bicep for managing Azure resources