Senior Data Engineer

Role Overview

• Design and build robust, scalable, and high-performance ETL/ELT data pipelines
• Develop and implement solutions using advanced programming languages (e.g., Python, SQL, DBT)
• Optimize and performance-tune complex data workflows and queries to handle massive data volumes and ensure minimal latency
• Assist with the design and implementation of data models for Data Warehouses (Snowflake etc.)
• Implement data quality frameworks (validation, monitoring, and testing) to ensure data accuracy, consistency, and completeness
• Establish comprehensive error logging and alerting systems for pipeline health and failure remediation
• Apply DevOps practices to data infrastructure, often involving Infrastructure as Code (e.g., Terraform)

Requirements

• 5+ years experience in the Data industry
• 4+ years experience in a Data Engineering role
• Expert in SQL to then implement where needed in Data Warehousing systems and Data Pipelines
• PostgreSQL / MySQL / Python / R / Redshift / Snowflake experience
• Experience in database design and database management
• A sound understanding of data management principles, including data quality assurance and data governance
• Experience in source code and version control systems
• Experience with infrastructure automation (Terraform)
• Familiarity working with containerized services (Docker and Kubernetes)
• Experience in the shipment of data using various methods, such as ETL/ELT, within tools such as Airflow (MWAA), Argo, dbt
• Experience using services within the AWS ecosystem (e.g, RDS, EC2, Redshift, EKS)
• Understanding or knowledge of Linux commands (bash scripting)
• Manages multiple projects, resolves challenges swiftly, delivers adaptable approaches, and values diverse opinions.
• Takes ownership, anticipates new problems, tests solutions, and mentors on effective problem-solving techniques.
• Sets quarterly goals aligned with team objectives, tracks progress in Workday, and communicates barriers to the manager. Takes personal responsibility for quality and timeliness of work, correcting mistakes and learning from them.
• Demonstrates expert understanding, introduces new tools/methodologies, and provides team training to enhance productivity.
• Tailors messages for audience understanding, engages respectfully in team meetings, synthesizes thoughts effectively, presents confidently, and actively listens to stakeholders.
• Consistently builds and maintains key stakeholder relationships, perceived as a trusted partner, offers sound guidance, and navigates challenging relationships effectively.

Tech Stack

• Airflow
• Amazon Redshift
• AWS
• Docker
• EC2
• ETL
• Kubernetes
• Linux
• MySQL
• Postgres
• Python
• SQL
• Terraform

Benefits

• 2 complimentary Getsmarter short courses per year
• Subsidised medical aid with Discovery Health Medical Scheme
• 4% 2U contribution towards Discovery Life Pension Fund and Group Risk Benefit
• Employee Assistance Program (EAP)
• Generous leave policy including time off to volunteer for non-profit organization, study leave, sports leave and a company-wide festive season break