SPECTRAFOR is seeking a skilled Data Engineer to support compliance reporting initiatives by building scalable, auditable, and high-quality data pipelines. This role focuses on developing Palantir Foundry-native solutions and ensuring regulatory-grade data accuracy and traceability while collaborating with compliance stakeholders and data teams.
Responsibilities:
- Design, build, and maintain Palantir Foundry pipelines for compliance reporting
- Develop data validation and data quality frameworks within Foundry
- Implement large-scale data transformations using PySpark and Spark SQL
- Build compliance-driven checks (data completeness, consistency, reconciliation, exception handling)
- Ensure data lineage, auditability, and reproducibility across all pipelines
- Collaborate with compliance SMEs to validate reporting logic and outputs
- Monitor pipeline performance, troubleshoot issues, and support data quality remediation
Requirements:
- Strong experience with Python
- Hands-on expertise in PySpark / Spark SQL
- Advanced proficiency in SQL
- Proven experience working with Palantir Foundry
- Experience building data validation / data quality frameworks
- Understanding of compliance, audit, or regulatory reporting requirements
- Strong problem-solving and analytical skills
- Experience with OEIS or regulatory compliance reporting
- Exposure to Foundry Workshop
- Experience with ontology-driven or curated datasets
- Background in utilities, infrastructure, or asset-based industries