Data Engineer (Mid Level)

Irth is building an AI-driven threat and risk management platform for pipeline asset integrity. The Data Engineer will develop ingestion, transformation, governance, and orchestration pipelines using Databricks, Spark, and modern lakehouse technologies to prepare customer data for analytics and machine learning.

Responsibilities:

• Pipeline Development — Primary Responsibility
• Build and maintain ingestion pipelines for structured and semi-structured sources, including GIS, inline inspection data, SCADA, maintenance systems, and enterprise systems of record
• Implement batch and streaming ingestion using Databricks Workflows, Spark, PySpark, SQL, and declarative pipeline tooling
• Apply medallion architecture patterns (Bronze, Silver, Gold) for transformation, standardization, and enrichment
• Implement change data capture (CDC), slowly changing dimensions (SCD), schema evolution, and data-validation rules
• Normalize third-party and public data feeds, including weather history, soil characteristics, satellite-derived data, and one-call ticket data, into the shared data model
• Build pipelines that automate normalization of units, schemas, and semantics across inconsistent customer data
• Implement automated data-quality repair workflows, including gap filling, error correction, and reconciliation, with clear provenance for every synthesized value
• Work with data scientists to productionize document-extraction pipelines that parse reports, spreadsheets, and field records into the target schema
• Build human-in-the-loop review and exception workflows so low-confidence extractions are surfaced rather than propagated silently
• Configure and manage Delta Lake tables, partitioning strategies, and optimization routines
• Implement metadata, lineage, and cataloging standards using Unity Catalog
• Build and maintain connectors to customer systems of record with configurable refresh cadences
• Support geospatial data processing, including spatial joins and alignment of results to pipeline centerline geometry
• Implement data-quality tests, profiling, and drift monitoring based on standards established by the Data Architect
• Apply access-control policies, security rules, and classification tags defined by the governance model
• Implement lineage capture sufficient to support regulatory traceability from ingestion through model output
• Build, schedule, and monitor data workflows, and own alerting and failure handling for the pipelines you develop
• Contribute to CI/CD for pipeline code, including version control, automated testing, and environment promotion
• Troubleshoot production incidents, recover failed pipeline runs, and optimize performance and infrastructure costs
• Work closely with the Data Architect to translate architectural designs into production implementations and identify gaps or ambiguities in the design
• Participate in architecture, design, and code reviews
• Document pipelines, transformation logic, data dictionaries, job schedules, operational procedures, and runbooks

Requirements:

• 3-5 years of experience in data engineering, ETL development, or cloud data platform engineering
• Hands-on experience with Databricks, Spark, PySpark, or comparable distributed data-processing technologies
• Strong SQL skills and experience with structured data transformation
• Experience with at least one major cloud platform; Azure experience preferred
• Familiarity with data modeling, data-quality practices, schema evolution, and pipeline troubleshooting
• Experience with workflow orchestration and scheduling frameworks
• Understanding of core data-security practices, including access control, encryption, and credential management
• Experience with Git-based development and comfort working within a code-reviewed engineering team
• Experience with Delta Lake, medallion architecture, and lakehouse engineering best practices
• Experience with Unity Catalog, Microsoft Purview, or comparable metadata and data-lineage tooling
• Experience building pipelines that ingest unstructured or semi-structured documents
• Experience with geospatial data processing and common GIS data formats
• CI/CD and DevOps experience for data workloads, including infrastructure as code (IaC)
• Experience preparing and transforming data specifically for machine learning or probabilistic model consumption
• Cloud or Databricks certifications
• Experience using AI-assisted coding tools such as Cursor or GitHub Copilot and/or agentic coding tools such as Claude Code as part of a professional development workflow
• Experience integrating oil and gas or utility asset data, including pipelines, facilities, and GIS assets, into a data platform
• Understanding of asset integrity concepts, including inspection data, risk scoring, corrosion, and defect tracking
• Familiarity with regulatory and compliance reporting requirements for pipeline or asset integrity data
• Experience migrating customers from legacy or spreadsheet-based systems to modern data platforms