Data Engineering Lead

YipitData is a leading market research and analytics firm specializing in the disruptive economy. They are seeking a Data Engineering Manager to lead the design and optimization of end-to-end data pipelines, collaborating with various stakeholders to deliver scalable data solutions.

Responsibilities:

• Own the design, build, and optimization of end-to-end data pipelines that power our vendor universe
• Establish and enforce best practices in data modeling, orchestration, and system reliability
• Collaborate with product, engineering, and business stakeholders to translate requirements into robust, scalable data solutions
• Work extensively with Databricks and Airflow for large-scale data processing and orchestration
• Troubleshoot and resolve complex pipeline issues to ensure reliability and performance
• Contribute to the team’s technical strategy, helping drive improvements in scalability, performance, and efficiency
• Lead, mentor, and support engineers through challenges, code reviews, and project execution

Requirements:

• 6+ years of professional experience in Data Engineering or equivalent technical roles (e.g., data architecture, big data development, or ETL engineering)
• 2+ years of managerial experience, including mentoring, team leadership, and supporting delivery
• Strong expertise in SQL and distributed data systems
• Proficiency with PySpark and Databricks for processing and scaling large datasets
• Hands-on experience with Airflow for pipeline orchestration (Dagster/dbt a plus)
• Proven track record of delivering in fast-paced, deadline-driven environments with minimal oversight
• Strong problem-solving skills and ability to translate business needs into scalable technical solutions
• Excellent communication and collaboration skills with both technical and non-technical stakeholders
• Experience leveraging AI/ML models, vector search, or Elasticsearch to enhance data pipelines
• Familiarity with Django or similar web frameworks to support data-driven applications