Data Engineer

Arbor TekSystems is seeking a Data Engineer with expertise in Databricks. The role involves owning the migration strategy and technical execution for moving legacy ETL workloads to the Databricks Lakehouse, while optimizing data workflows and establishing best practices.

Responsibilities:

• Design the target Databricks Lakehouse architecture utilizing Delta Lake, Photon, and Unity Catalog
• Establish global code refactoring standards, optimization benchmarks, and PySpark best practices
• Resolve highly complex dependency mappings and architect seamless, zero-downtime dual-run strategies
• Lead the technical deployment and integration of specialized migration accelerators
• Review automated output from migration tools and manually refactor complex legacy logic into high-performing PySpark notebooks
• Eliminate legacy anti-patterns such as massive row-by-row processing and inefficient lookups
• Optimize PySpark code performance using advanced Spark features including Z-Ordering, partitioning, and caching
• Build robust Databricks Workflows and orchestrate complex DAGs based on comprehensive source lineage

Requirements:

• Experience with Databricks and Delta Lake
• Knowledge of Unity Catalog and Photon
• Proficiency in DataStage
• Strong skills in PySpark, Python, SQL, and Shell Scripting
• Experience with AWS and CI/CD deployment pipelines
• Familiarity with Apache Airflow and Databricks Workflows
• Ability to design target Databricks Lakehouse architecture
• Experience in establishing global code refactoring standards and optimization benchmarks
• Capability to resolve complex dependency mappings
• Experience in leading technical deployment and integration of migration accelerators
• Hands-on experience in reviewing automated output from migration tools
• Ability to refactor complex legacy logic into high-performing PySpark notebooks
• Experience in optimizing PySpark code performance using advanced Spark features
• Ability to build robust Databricks Workflows and orchestrate complex DAGs