Data Engineer III

Role Overview

• Design, implement, and optimize ETL and ELT pipelines using dbt to extract, transform, and load data into the data warehouse.
• Build, evolve, and maintain infrastructure for optimal data transformation and integration using Azure Event Hub or Event Grid, PostgreSQL, and Azure Cosmos DB.
• Manage and monitor complex data processing and storage infrastructure, including Data Lakes leveraging Apache Iceberg table formats.
• Perform advanced manipulation and analysis of large datasets using Trino for distributed SQL querying across diverse data sources.
• Utilize Python for high-performance data manipulation, scripting, and process automation.
• Containerize and orchestrate data workloads using Docker and Kubernetes to ensure scalability and reliability.
• Develop architectural frameworks and tools that enable Data Science teams to build, train, and scale machine learning models on the Azure Machine Learning Platform.
• Automate and optimize data processes while implementing robust logging and monitoring practices.
• Partner strategically with Analytics and cross-functional teams to deliver scalable data solutions within agreed timelines.
• Monitor, troubleshoot, and remediate complex data quality issues, including database schema modifications.
• Support data governance initiatives, including cataloging and lineage tracking.
• Collaborate with senior engineers to design data models, develop summarized datasets, and build data APIs with minimal guidance.
• Coach and mentor junior engineers on complex technical challenges.
• Write scripts and queries to support ad hoc data analysis.
• Perform other duties as assigned.

Requirements

• 5+ years of relevant data engineering experience.
• A Bachelor’s degree in a related field.
• Expert-level SQL and advanced experience with PostgreSQL schema design and query optimization.
• Deep proficiency in dbt for developing and managing modular, documented ETL and ELT pipelines.
• Advanced Python skills, including experience with pandas and polars for high-performance data manipulation.
• Hands-on experience with Azure Event Hub or Event Grid subscriptions and Azure Cosmos DB for streaming data storage.
• Experience managing Data Lake environments using Apache Iceberg table formats.
• Proficiency with Trino for federated SQL queries across disparate data sources.
• Experience with Git and modern source code management practices.
• Strong understanding of data management systems, data warehousing, and data modeling concepts.
• Ability to communicate effectively with both technical and non-technical stakeholders.

Tech Stack

• Apache
• Azure
• Docker
• ETL
• Kubernetes
• Pandas
• Postgres
• Python
• SQL

Benefits

• Comprehensive medical, vision and dental benefits that give you peace of mind.
• Flexible Spending Accounts (FSA) that let you use pre-tax dollars to cover healthcare expenses.
• A fantastic 401K with matching contributions that helps you plan for retirement and build wealth over time.
• Generous sick, vacation, and holiday benefits that give you the time and flexibility you need to enjoy life.
• A gym membership contribution that supports your well-being, and helps you stay energized and focused.
• An internal referral program that rewards you for bringing talented people to the team.
• Company events that foster a positive and inclusive culture, and create opportunities to bond and grow with your colleagues.