Senior Databricks Data Engineer, Spreadsheet & Flat-File Sources

Lumenalta partners with global enterprises to design and build large-scale data platforms that power critical products and operations. The Senior Databricks Data Engineer will inventory spreadsheet and flat-file sources, build governed Databricks ingestion pipelines and Delta tables, and integrate the data with engineering systems. The role also includes implementing data quality, lineage, schema evolution, business-rule translation, and documentation.

Responsibilities:

• Inventory the Excel workbooks and file exports that business and engineering teams rely on, and document their structure, owners, and downstream use
• Build Auto Loader pipelines to ingest workbooks and flat files landed in S3, with explicit schemas and typed Bronze and Silver tables
• Parse multi-sheet, inconsistently structured workbooks at scale: shifting header rows, merged cells, embedded totals, mixed types in a column, and layouts that differ between versions of the same file
• Configure schema evolution in Databricks declarative pipelines, including new column handling, rescued data columns, and controlled failure on renamed columns
• Implement data quality expectations that catch the failure modes these sources produce, such as duplicate rows, blank keys, and silently changed formats
• Conform ingested data to the platform's part number plus revision keying so it joins to BOM and engineering data from other sources
• Ensure lineage is captured for every pipeline, orchestrate with Databricks Workflows, and document each source for handoff
• Work with the business owners of each workbook to confirm meaning, business rules, and what "correct" looks like
• Translate spreadsheet logic, such as formulas, lookups, and pivot tables, into maintainable SQL and documented business rules
• Validate that platform outputs match the numbers users trust today, and explain differences when they do not
• Document datasets, metrics, and definitions so reporting stays consistent after handoff

Requirements:

• Production experience building ingestion pipelines on Databricks
• Hands-on experience ingesting Excel and flat-file sources into Spark or Databricks at scale and the judgment to know when each is appropriate
• A track record of making human-maintained data reliable: shifting headers, merged cells, mixed types, and layout drift between versions
• Experience with Auto Loader for incremental file ingestion from S3
• Experience with Databricks declarative pipelines or Delta Live Tables, including schema evolution and data quality expectations
• Experience conforming ingested data to shared keys so it joins with data from other systems
• Strong PySpark and SQL, and working knowledge of Delta Lake and medallion architecture
• Experience working directly with non-technical data owners to pin down definitions and business rules
• US citizenship
• Fluent English, both written and spoken
• This position supports a U.S. Government contract that requires all personnel to possess U.S. Citizenship
• Experience extracting from Microsoft Access databases (ODBC, JDBC, or file-level tooling)
• Prior work with manufacturing or supply chain data, especially BOMs, parts, revisions, and supplier records
• Familiarity with PLM or ERP data, such as Teamcenter or Oracle E-Business Suite
• Prior work in a defense, aerospace, or FedRAMP environment, or with CUI or ITAR-controlled data
• Databricks Data Engineer Professional certification