Senior Data Engineer

The confidential company’s Core Data team builds and scales data platforms and analytics capabilities that support decision-making across its vehicle domain. The Senior Data Engineer will migrate the Vehicle Catalog to a new Databricks environment by refactoring ETL/ELT and analytics pipelines, migrating datasets, validating data parity, and supporting related dashboard and reporting migrations.

Responsibilities:

• Work with one of our Staff Data Engineers to understand the migration plan, sequence your work against it, and provide regular progress updates
• Refactor existing ETL/ELT code written in PySpark, DABs and dbt from the current Databricks environment to meet the patterns and standards of the new account
• Migrate data pipelines and their underlying datasets, including backfills, and validate parity between source and target
• Build and run reconciliation checks — row counts, schema conformance, freshness, completeness — to confirm migrated data matches the legacy environment
• Migrate dashboards and downstream reporting assets to point at the new datasets, coordinating with the analysts who own them
• Update orchestration, scheduling, alerting, and runbooks so migrated pipelines are supportable after cutover
• Troubleshoot pipeline failures and data discrepancies during and after cutover
• Document what was migrated, what changed, and anything the team needs to know to own it going forward
• Take on adjacent migration and data engineering requests as the program evolves

Requirements:

• 6+ years of experience in Data Engineering, Analytics Engineering, or Software Engineering working with production data systems
• Strong expertise in SQL and Python for building scalable data pipelines and transformations
• Strong familiarity with AI tools like Claude Code / Devin for SW development process
• Hands-on experience building ELT pipelines using modern cloud data platforms (Databricks experience required)
• Building robust testing frameworks for Data Migration ( Completeness, Quality etc)
• Deep experience with dbt, including model development, testing, documentation, and CI/CD integration
• Experience designing and operating production-grade data pipelines with monitoring and observability
• Strong collaboration skills and ability to partner with engineering teams, analysts, and operational stakeholders
• Bachelor's or Master's degree in Computer Science, Engineering, Mathematics, or related field, or equivalent practical experience
• Prior data or platform migration experience
• Familiarity with Fivetran connector management and ingestion architecture
• Experience with data contracts and schema governance
• Knowledge of streaming or near-real-time data pipelines