Senior Data Engineer

Truthset is a data intelligence company that validates consumer data accuracy for marketing and data-driven decision-making. The Senior Data Engineer will own and scale the data platform supporting accuracy, audience, and identity products, including provider ingestion, privacy compliance, PostgreSQL-to-Databricks migration, and large-scale performance optimization. The role also leads data governance, quality, production engineering, and technical standards.

Responsibilities:

• Own provider and partner data pipelines — onboard new data providers and partners, and operate a growing set of ingestion pipelines with clear schemas, data contracts, validation, and SLAs
• Build and run compliance and opt-out pipelines — design auditable pipelines that process consumer deletion, opt-out, and suppression requests (e.g., California Delete Act/DROP, CCPA/CPRA, other state privacy laws) and propagate them reliably across raw, derived, and delivered datasets, including verifiable physical purge in Delta Lake and cloud storage
• Migrate PostgreSQL workloads to Databricks — lead the re-architecture of application-level Postgres functionality (e.g., identity resolution, matching, and reporting) into Spark/Delta, designed for distributed execution rather than lift-and-shift, with parity testing and safe cutover
• Optimize performance and cost at scale — tune Spark and Delta workloads over billions of rows: data layout (partitioning, liquid clustering, Z-ordering), join strategy, skew, file sizing, caching, and cluster/serverless configuration
• Model data for our products — design lakehouse data models (medallion architecture) that support accuracy scoring, audience generation, identity graphs, and provider attribution
• Govern and share data — manage access, lineage, and cataloging in Unity Catalog, and support secure data exchange with partners via Delta Sharing and other delivery mechanisms
• Ensure data quality and reliability — implement automated quality checks, monitoring, and alerting; lead troubleshooting and root-cause analysis for data incidents
• Support outbound deliveries — build and maintain reliable data feeds to activation and marketplace partners
• Engineer for production — write clean, tested, well-documented Python and SQL; deploy through CI/CD and infrastructure-as-code (e.g., Databricks Asset Bundles, Terraform)
• Lead technically — shape architecture decisions, set engineering standards, review code, mentor others, and partner closely with data science, product, and data operations

Requirements:

• 6+ years of experience in data engineering or a closely related role, including ownership of production pipelines at large scale
• 2+ years of hands-on production experience with Databricks
• Expert-level Spark (PySpark and Spark SQL), Python, and SQL
• Deep working knowledge of Delta Lake — MERGE/upserts, change data feed, schema evolution, time travel, OPTIMIZE/VACUUM, and deletion vectors
• A track record of diagnosing and fixing performance problems on datasets with hundreds of millions to billions of rows
• Strong PostgreSQL skills (query plans, indexing, tuning) and experience migrating relational/application workloads to a distributed platform
• Experience ingesting and operating many external data feeds from third parties with varied formats, delivery methods (S3, SFTP, data sharing), and quality levels
• Hands-on experience implementing privacy compliance in data systems — deletion, suppression, retention, and auditability under CCPA/CPRA, GDPR, or similar
• Solid AWS experience (S3, IAM, multi-account environments) and orchestration experience (Databricks Workflows/Jobs, Airflow, or similar)
• Strong Git, testing, and CI/CD practices
• Clear communication, sound judgment, and comfort owning outcomes in a small team where priorities evolve quickly
• Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience
• Identity resolution experience — hashed emails (HEMs), device IDs, identity graphs, and entity resolution/matching at scale
• Ad-tech or mar-tech background — audience activation through DSPs/SSPs and data marketplaces, measurement and impression-log data, or data clean rooms
• Unity Catalog administration, Delta Sharing, Lakeflow/Delta Live Tables, Databricks Asset Bundles, or Terraform
• Experience with other cloud data platforms, such as Snowflake
• Databricks Certified Data Engineer Professional (or equivalent)
• Experience using AI-assisted development tools effectively in a production engineering workflow
• Experience at an early-stage or growth-stage startup