Lead Data Engineer

STAFFXPERT LLC is seeking a Lead Data Engineer to support and enhance their data engineering processes. The primary responsibilities include building and maintaining data pipelines, ensuring data quality, and collaborating with various teams to meet data requirements.

Responsibilities:

• Serve as L3 support: triage high-severity incidents, perform advanced debugging/root-cause analysis, deploy hotfixes, and create runbooks for L2 teams
• Build and maintain batch/streaming data pipelines using ETL/ELT tools (dbt,) to integrate and transform multi-source data
• Implement data quality validation, monitoring, alerting, and documentation; optimize pipelines for performance, cost, and reliability (partitioning, indexing, error handling)
• Partner with analytics, data science, and business teams to deliver data requirements, troubleshoot issues, and ensure SLAs for freshness/completeness

Requirements:

• 8–10+ years data engineering experience building and supporting production pipelines at scale
• Design, build, and maintain data ingestion, transformation, and delivery pipelines across structured and semi-structured data sources
• Develop modular, reusable data transformation logic using Python, SQL, and frameworks such as dbt
• Implement data models and schemas optimized for analytics and reporting (star, snowflake, or dimensional)
• Apply Medallion Architecture principles to organize data layers for quality, traceability, and performance
• Use cloud-native data services such as AWS Glue, S3, Redshift, EMR or Azure Data Factory, ADLS, Synapse to manage data workflows
• Set up and manage data pipeline orchestration, scheduling, and monitoring using Airflow, ADF, or equivalent tools
• Apply data quality checks, validation logic, and logging mechanisms to ensure consistency and trust in data assets
• Collaborate with analysts, scientists, and architects to design data models that align with business and analytical needs
• Maintain code versioning, testing, and CI/CD standards for data pipeline development
• Proven cloud data platform + orchestration experience (Snowflake/Big Query + Airflow/dbt)
• L3 support experience: incident management, on-call rotations, debugging distributed data systems
• Strong understanding of data engineering fundamentals — ETL/ELT design, data modelling, schema evolution, and data integrity
• Proficient in Python and SQL for data transformation, automation, and workflow scripting
• Hands-on experience with cloud-based data services in AWS (S3, Glue, Redshift, EMR) or Azure (ADLS, ADF, Synapse)
• Working knowledge of distributed data processing concepts (Spark, Hive, or equivalent)
• Familiarity with dbt for transformation design, testing, and data documentation
• Awareness of Medallion Architecture and data layering concepts for scalable data management
• Understanding of orchestration frameworks like Airflow or Data Factory for scheduling and monitoring pipelines
• Knowledge of Git-based version control, CI/CD, and basic DevOps practices in data workflows
• Have an AI skill set, a little bit on Claude, ChatGPT, and other tool supports, or at least who can pick up those skills