Data Engineer

SPECTRAFOR is seeking a skilled Data Engineer to support compliance reporting initiatives by building scalable, auditable, and high-quality data pipelines. This role focuses on developing Palantir Foundry-native solutions and ensuring regulatory-grade data accuracy and traceability while collaborating with compliance stakeholders and data teams.

Responsibilities:

• Design, build, and maintain Palantir Foundry pipelines for compliance reporting
• Develop data validation and data quality frameworks within Foundry
• Implement large-scale data transformations using PySpark and Spark SQL
• Build compliance-driven checks (data completeness, consistency, reconciliation, exception handling)
• Ensure data lineage, auditability, and reproducibility across all pipelines
• Collaborate with compliance SMEs to validate reporting logic and outputs
• Monitor pipeline performance, troubleshoot issues, and support data quality remediation

Requirements:

• Strong experience with Python
• Hands-on expertise in PySpark / Spark SQL
• Advanced proficiency in SQL
• Proven experience working with Palantir Foundry
• Experience building data validation / data quality frameworks
• Understanding of compliance, audit, or regulatory reporting requirements
• Strong problem-solving and analytical skills
• Experience with OEIS or regulatory compliance reporting
• Exposure to Foundry Workshop
• Experience with ontology-driven or curated datasets
• Background in utilities, infrastructure, or asset-based industries