Design and implement scalable, efficient data pipelines and infrastructure to support business intelligence and analytics initiatives (Databricks, Power Automate).
Lead data engineering to build robust, high-performance data systems that align with business objectives as well as enabling fit for GenAI and Machine Learning datasets.
Collaborate with stakeholders across the organization to understand data requirements and ensure data solutions meet business needs and serve appropriate data.
Oversee the development, optimization, and maintenance of data pipelines, ensuring data is collected, processed, and made available for analysis in a timely and accurate manner.
Ensure data quality, integrity, and governance across all data systems by implementing best practices for data validation, security, and privacy (Unity Catalog in Databricks).
Develop and maintain ETL processes to integrate data from various sources into centralized data warehouses and data lakes.
Partner with analysts and business teams to design data architectures that enable effective reporting, analysis, and decision-making.
Act as the primary point of contact for data engineering, ensuring smooth communication between technical teams and business stakeholders.
Translate business needs into technical specifications, ensuring the data infrastructure supports both current and future analytics requirements.
Monitor and optimize the performance of data systems and pipelines, ensuring they meet service level agreements (SLAs) and business expectations.
Continuously evaluate and implement new technologies and tools to enhance data processing capabilities and improve overall system performance (Zapier, Genie AI).
Continuously identify areas for process improvements and implement automation to enhance the efficiency and scalability of data workflows.
Ensure data storage and processing solutions are optimized for cost and performance, adapting to evolving business needs.
Requirements
5+ years of experience in data engineering, data warehousing, or a similar role.
Proven experience in designing, building, and maintaining large-scale data systems and workflows in cloud environments.
Expertise in SQL and Python, or other programming languages used for data processing and pipeline development.
Strong knowledge of data warehousing solutions (e.g., Snowflake, Databricks, Redshift, BigQuery) and cloud platforms (e.g. AWS, GCP, Azure).
Experience with ETL tools, data integration platforms, and data pipeline orchestration tools (e.g. Apache Airflow, Talend, dbt).
Knowledge of reporting and Business Intelligence Tools (e.g. Power BI, Tableau, Cognos)
Familiarity with data governance principles, data security, and privacy standards.
Tech Stack
Airflow
Amazon Redshift
Apache
AWS
Azure
BigQuery
Cloud
Cognos
ETL
Google Cloud Platform
Python
SQL
Tableau
Unity
Senior Data Engineer at My Money Matters | JobVerse