Key skills
PySparkDatabricks
About this role
Role Overview
- Manage and optimise the Databricks platform to ensure high performance and cost efficiency
- Analyse DBU consumption and drive improvements in platform utilisation
- Monitor and enhance PySpark pipeline performance
- Make informed decisions on scaling up/down clusters based on workload demands
- Establish and maintain automated release management processes
- Focus on operational excellence across the platform environment
- Gather performance data and provide clear evidence-based insights to stakeholders and Databricks teams
Requirements
- Strong hands-on experience with Databricks platform engineering
- Solid expertise in PySpark and performance tuning
- Proven experience optimising data pipelines and platform costs
- Experience with cluster management and scaling strategies
- Ability to analyse performance metrics and present findings effectively
- Familiarity with platform operations and reliability best practices
Tech Stack
- PySpark