Lead Data Engineer

Redhorse Corporation is seeking a Senior Data Engineer / Lead to support the US National Defense Strategy. The role translates defense requirements into scalable data pipelines and actionable insights, while providing technical leadership, developing and optimizing data platforms, and collaborating with government stakeholders and technical teams.

Responsibilities:

• Technical Leadership & Task Oversight: Provide day-to-day task prioritization and technical direction to team members; lead code reviews, ensure design best practices, and mentor engineers in modular Python development and SQL optimization
• Requirements Development & Architecture: Work directly with government stakeholders to elicit, refine, and translate complex mission requirements into robust technical specifications, interface designs, and scalable data models
• Pipeline & Platform Development: Architect, develop, and optimize high-throughput ETL/ELT workflows using Databricks; maintain, refactor, and enhance existing Python and SQL codebases for performance, stability, and security
• Pipeline Diagnostics & Troubleshooting: Identify pipeline bottlenecks, debug complex data workflows, and implement automated solutions to enhance pipeline reliability and data integrity
• Cross-Functional Collaboration: Partner closely with government product owners, data scientists, Data Analysts, and technical peers to document architecture, coordinate technical sprints, and communicate engineering trade-offs to technical and non-technical audiences
• Technical Documentation: Develop new and maintain existing pipeline architecture documentation relating to pipeline code architecture and configurations to ensure high system reliability and seamless team onboarding

Requirements:

• Must be a U.S. Citizen with a current, active U.S. Government Secret clearance
• Minimum of 8+ years of professional experience in data engineering, building enterprise-scale data platforms and pipelines
• Demonstrated experience providing technical mentorship, performing code reviews, and providing day-to-day technical task oversight to other engineers
• Proven, hands-on experience designing and optimizing data transformation workflows in Databricks
• Advanced skills in modular, object-oriented Python/PySpark scripting and extensive experience writing, tuning, and optimizing complex SQL queries
• Proven experience building functional, user-centric reports and dashboards using modern data visualization tools
• Hands-on experience with Apache Spark and distributed computing frameworks
• Familiarity with modern agile and DevOps tools such as GitLab Jira
• Experience working with DoD, defense, or intelligence community data environments and compliance standards
• Knowledge of data governance, metadata management, and data cataloging frameworks
• Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or a related technical discipline