Lead Data Engineer

Black Canyon Consulting LLC is hiring a Lead Data Engineer to design and implement the core analytical data platform for a federal permitting modernization initiative. The role leads development of a cloud-based lakehouse, data engineering patterns, data quality and reconciliation processes, and AI-enabled workflows while guiding other Data Engineers.

Responsibilities:

• Design and implement the core DuckDB / Apache Iceberg data platform on AWS
• Define physical data organization, schema, partitioning, file layout, snapshot, compaction, and optimization strategies for Iceberg tables
• Integrate DuckDB with Amazon S3, Apache Iceberg, and AWS Glue Catalog
• Establish bronze, silver, and gold data-layer conventions and reusable engineering patterns
• Lead and mentor Data Engineers, providing technical direction, code reviews, and engineering guidance
• Build and maintain analytical datasets supporting application services, reporting, analytics, and data science
• Develop reusable patterns for incremental loads, CDC, schema evolution, late-arriving data, backfills, and historical reconstruction
• Work with the Data Architect to implement the common canonical data model and source-to-canonical mappings
• Lead migration and harmonization of datasets from multiple source systems
• Implement reconciliation processes that demonstrate completeness and accuracy between source and target systems
• Establish automated data-quality monitoring, operational observability, and testing
• Implement agentic and AI-enabled data engineering patterns supporting activities such as source analysis, mapping, pipeline generation, troubleshooting, reconciliation, and quality assurance
• Support integration of structured data with associated document and geospatial datasets
• Lead development of certified data products, analytical datasets, and business metrics
• Optimize DuckDB queries and Iceberg datasets for performance and efficiency
• Develop engineering documentation, operational procedures, and technical handoff materials

Requirements:

• Bachelor's degree in Computer Science, Data Engineering, Information Systems, Engineering, or a related field
• 6+ years of relevant data engineering or data platform experience, including experience leading or mentoring other engineers
• Strong hands-on experience with DuckDB and Apache Iceberg
• Advanced SQL and strong Python development experience
• Experience building cloud-based analytical platforms using object storage and columnar formats such as Parquet
• Strong understanding of partitioning, schema evolution, snapshots, compaction, incremental processing, and analytical query optimization
• Experience implementing bronze/silver/gold or comparable layered data architectures and transforming heterogeneous enterprise datasets
• Experience with AWS data services, particularly Amazon S3 and AWS Glue
• Experience designing reusable, metadata-driven ingestion and transformation frameworks
• Experience implementing automated data quality, reconciliation, monitoring, and operational controls
• Experience with agentic or AI-enabled data engineering solutions, including the use of agents or AI-assisted workflows for activities such as data mapping, pipeline generation, validation, testing, or troubleshooting
• Hands-on experience with agentic frameworks such as LangChain, LangGraph, or similar frameworks for building tool-using or multi-step AI workflows
• Strong troubleshooting, performance tuning, and technical leadership skills
• Experience with Kafka or Amazon MSK and CDC architectures
• Experience with GeoParquet, PostGIS, or geospatial datasets
• Experience with OpenSearch, document stores, or vector search technologies
• Experience integrating enterprise SaaS platforms, APIs, databases, and file-based sources
• Experience working in the U.S. Federal Government space
• Experience with permitting, regulatory, land management, environmental, energy, or natural-resource datasets