Forward Deployed Data Engineer IV, Databricks

E Source is a research, data analytics, and technology-focused professional services firm serving the utility industry in North America. The Forward Deployed Data Engineer IV will embed with utility clients to lead end-to-end data platform engagements, from technical discovery and architecture through deployment and handover, using Databricks, Spark, Python, SQL, and AWS. The role also involves building governed, reliable data systems, operator-facing applications, and reusable engineering frameworks.

Responsibilities:

• Embed with utility clients to design and build production data platforms end to end: ingestion, transformation, orchestration, quality, governance, and serving, using Databricks, Spark, Python, SQL, and Amazon Web Services (AWS)
• Own the technical architecture of each engagement and defend it to client architects, security teams, and platform owners
• Run technical discovery independently, separating the problem as stated from the underlying need, and say so when the requested solution is the wrong one
• Deliver a working, useful system early in each engagement, typically within the first few weeks, then iterate toward production hardening. Tactical solutions are legitimate; undocumented or unowned ones are not
• Lead migrations from legacy warehouses and on-premises systems to modern platforms, including undocumented dependencies
• Model data so that downstream teams can use it without re-deriving the same logic, using dimensional, lakehouse and medallion, and graph or semantic models where the domain calls for it
• Integrate with utility source systems such as asset and work management, historians and supervisory control and data acquisition (SCADA), customer information systems (CIS), advanced metering infrastructure (AMI), geographic information systems (GIS), and enterprise resource planning (ERP) through application programming interfaces (APIs), change data capture, file drops, and whatever is actually available
• Make pipelines observable and operable: lineage, quality checks, alerting, and runbooks the client’s engineers can act on
• Build and deploy operator-facing application layers, such as data quality and reconciliation dashboards, review and correction interfaces, and self-service data tools, so the platform is usable by client business teams and not just by their engineers. Databricks Apps is the preferred approach where the client runs Databricks
• Manage scope, timeline, and expectations against a defined statement of work, raising risk early
• Contribute reusable frameworks, ingestion patterns, and reference architectures that scale across clients, and provide product and implementation feedback to E Source engineering and product teams
• Apply data governance, security, privacy, and retention controls appropriate to the utility regulatory context

Requirements:

• • Bachelor's degree in computer science, information technology, or a related field
• • Five or more years of experience in data engineering, data platform, or analytics engineering, with at least one platform owned in production
• • Expert proficiency in Python, SQL, Databricks, and Spark, including Spark runtime internals and the ability to diagnose why a job is slow rather than adding compute
• • Experience building cloud-based data pipelines in AWS; working knowledge of a second cloud ecosystem preferred
• • Sound handling of the recurring hard problems: incremental versus full rebuild, idempotency, late-arriving data, deduplication, backfill strategy, and schema evolution
• • Batch and streaming pipeline design, with informed judgment about which the problem actually needs
• • Knowledge of data governance, security, privacy, lineage, and quality practices
• • Proficiency with Git, Docker, continuous integration and continuous delivery (CI/CD) tools, and at least one modern orchestration framework
• • Demonstrated experience using AI coding assistants and agentic development tools to rapidly prototype, test, and iterate on working software, compressing the path from idea to running system. This is a top skill set for the role
• • Experience in the energy or utility industry, or another asset-heavy sector, with working familiarity of the systems, data, and operational constraints utilities live with
• • Strong written communication and whiteboarding skills
• This role will be remote-based within the US, with significant travel to client sites, generally 30 to 50% and varying by engagement, including extended on-site periods during discovery, deployment, and go-live
• Applicants must be authorized to work for any employer in the US. We're unable to sponsor or take over sponsorship of employment visas at this time
• • Hold a master's degree in a relevant science, technology, engineering, or mathematics (STEM) field
• • Have graph database and knowledge graph modeling experience
• • Have run streaming platforms (Kafka or equivalent) and change data capture tooling in production
• • Have built feature pipelines and served data to machine learning (ML) and artificial intelligence (AI) systems
• • Have built and deployed Databricks Apps, or comparable operator-facing applications on a modern frontend framework (React, Streamlit, or Dash) with REST or GraphQL APIs
• • Have optimized cost and performance in Databricks: partitioning, file layout, clustering, caching, and compute right-sizing
• • Hold a Databricks certification
• working knowledge of a second cloud ecosystem preferred