Senior Data Engineer

Omni Federal is seeking a Senior Data Engineer to support the Air Force in developing and operating modernized applications. The role focuses on building and operating data pipelines, onboarding and validating data, and enabling discoverability and consumption through the program’s data lakehouse and enterprise data mesh.

Responsibilities:

• Design, build, and operate streaming (Debezium change-data-capture into Amazon MSK) and batch (Apache Spark via AWS EMR) data pipelines that rapidly onboard new data sources into the program’s Apache Iceberg-based data lakehouse, from initial ingestion through validation and production release
• Corroborate newly onboarded data against existing data lakehouse holdings to validate quality, consistency, and lineage before it is exposed to downstream consumers
• Develop and maintain the metadata, cataloging (AWS Glue as the Iceberg data catalog), and API Gateway integrations that make program data easily discoverable and consumable via Amazon Athena, Spark, and other Iceberg-compatible tools by downstream mission partners and vendors through the enterprise data mesh
• Own assigned technical tasking independently — translate well-articulated requirements into a working solution
• Engage directly with data providers and data consumers to understand the operational context behind each data source and translate that context accurately for both technical and non-technical audiences
• Participate in an agile (Kanban/Scrum) delivery team, contributing to sprint planning, technical design discussions, and pipeline/code reviews
• Maintain pipeline and infrastructure code as Infrastructure-as-Code (Terraform) and follow established DevSecOps build and deployment pipeline processes
• Support data governance, quality, and auditability practices in accordance with Government-approved data standards for the platform

Requirements:

• Active DoD Secret Clearance
• 8+ years of professional data engineering experience, including hands-on design and operation of production data pipelines
• Demonstrated experience with streaming and batch data pipeline technologies, particularly Debezium (change-data-capture), Amazon MSK (Kafka), Apache Spark (AWS EMR), and Apache Airflow
• Strong proficiency in Python and SQL, with experience integrating and validating data from multiple heterogeneous sources
• Experience with AWS data services (e.g., S3, Glue, Aurora) and containerized environments (Kubernetes), delivered through DevSecOps/CI-CD pipelines
• Demonstrated ability to work independently against well-defined tasking with minimal oversight, and to communicate effectively with both technical and non-technical stakeholders, including data providers and consumers
• Experience supporting DoD or Intelligence Community programs, particularly Platform One or other DoD software factory environments
• Familiarity with container security and software supply chain concepts (SBOMs, vulnerability scoring and attestation)
• Experience with the Apache Iceberg table format or similar open table formats (e.g., Delta Lake, Hudi) for lakehouse architectures
• Experience with event-driven architecture (e.g., AWS EventBridge) and Infrastructure-as-Code (Terraform)
• Experience designing or consuming API Gateways in an enterprise data mesh architecture
• Prior experience in a vendor- or customer-facing data steward or data liaison capacity