Senior Data Engineer - Databricks

Dataworks is a US-based investment platform focused on energy assets, seeking a Senior Data Engineer Contractor to enhance their existing production data platform. The role involves ensuring data reliability, quality, and production discipline within an Azure-hosted environment, while collaborating with business stakeholders to deliver high-quality datasets for analytics.

Responsibilities:

• Productionizing ingestion from external vendors into structured, reliable datasets
• Implementing incremental processing, safe backfills, and reproducible historical replays
• Building and maintaining curated “model-ready” datasets with stable keys
• Designing and enforcing data quality gates that prevent bad publishes
• Ensuring deterministic aggregations and preserving time-series directionality
• Establishing run discipline: manifests, run IDs, lineage, rerunnable workflows
• Improving orchestration, monitoring, retries, and alerting
• Partnering with analytics stakeholders to define stable data contracts

Requirements:

• Has built and maintained production-grade data pipelines in the cloud
• Understands incremental processing, schema evolution, partitioning, and optimization
• Has implemented data quality frameworks and publish gates
• Is comfortable being highly autonomous
• Can operate independently while participating in stand-ups and team alignment
• Has experience in energy, commodities, logistics, or flow-based systems (strong plus)
• East Coast US timezone overlap preferred