Data Engineer

Yochana is seeking a Databricks Data Engineer who will own the end-to-end migration strategy to the Databricks Lakehouse. This role involves designing target architecture, optimizing data workflows, and ensuring a seamless transition from legacy systems.

Responsibilities:

• Design the target Databricks Lakehouse architecture utilizing Delta Lake, Photon, and Unity Catalog
• Establish global code refactoring standards, optimization benchmarks, and PySpark best practices
• Resolve highly complex dependency mappings and architect seamless, zero-downtime dual-run strategies
• Lead the technical deployment and integration of specialized migration accelerators
• Review automated output from migration tools and manually refactor complex legacy logic into high-performing PySpark notebooks
• Eliminate legacy anti-patterns such as massive row-by-row processing and inefficient lookups
• Optimize PySpark code performance using advanced Spark features including Z-Ordering, partitioning, and caching
• Build robust Databricks Workflows and orchestrate complex DAGs based on comprehensive source lineage

Requirements:

• Design the target Databricks Lakehouse architecture utilizing Delta Lake, Photon, and Unity Catalog
• Establish global code refactoring standards, optimization benchmarks, and PySpark best practices
• Resolve highly complex dependency mappings and architect seamless, zero-downtime dual-run strategies
• Lead the technical deployment and integration of specialized migration accelerators
• Review automated output from migration tools and manually refactor complex legacy logic into high-performing PySpark notebooks
• Eliminate legacy anti-patterns such as massive row-by-row processing and inefficient lookups
• Optimize PySpark code performance using advanced Spark features including Z-Ordering, partitioning, and caching
• Build robust Databricks Workflows and orchestrate complex DAGs based on comprehensive source lineage
• Core Platforms: Databricks, Delta Lake, Unity Catalog, Photon, DataStage
• Languages & Frameworks: PySpark, Python, SQL, Shell Scripting
• Cloud & DevOps: AWS alongside CI/CD deployment pipelines
• Orchestration: Apache Airflow, Databricks Workflows