Note: Need solid Architect level candidate with good communication skills, this is a customer facing role.
Required Skills: Good customer presence and ability to answer a variety of Databricks questions.
GitHub and CI/CD experience is a plus Day-to-day
Responsibilities:
Design, develop, and maintain data engineering pipelines in Databricks using PySpark or SQL
Experience developing streaming pipelines using Databricks Autoloader or Structured Streaming
Data Modeling experience designing Delta Lake tables and the Bronze-Silver-Gold Medallion Architecture
Understanding of Spark for performance optimizations including Delta Lake optimizations
Experience with cloud service provider (Azure) services used in Databricks’ implementations (e.g., Azure ALDS, IAM)
Triaging ad hoc questions across the Databricks product suite Certifications
Preferred Skills: FiveTran
Responsibilities: Support ETL and CI/CD pipelines to create tables in the data lake.
Support data refresh and reconciliation processes to validate data accuracy.
Expected Deliverable(s): Data pipelines, tables, row-level security configurations, and ETL processes to automatically update datasets daily
