Databricks Engineer

An unnamed Fortune 500 utilities organization is hiring a Senior Databricks Engineer for a customer platform modernization initiative. The role focuses on designing, developing, and optimizing data pipelines and integration frameworks to consolidate customer data from legacy and enterprise systems into a Databricks Lakehouse.

Responsibilities:

• Design, develop, and maintain enterprise-scale data engineering solutions using the Databricks Lakehouse Platform
• Support the modernization of the organization's existing customer data platform, migrating legacy data infrastructure and integration processes into Databricks
• Build and optimize complex ETL/ELT pipelines using Python, PySpark, Spark SQL, and Delta Lake
• Develop scalable data ingestion frameworks to integrate customer information from multiple internal and external enterprise systems
• Consolidate and transform customer data from disparate legacy platforms into a consistent, unified data environment
• Design and implement data models supporting customer profiles, account information, service interactions, billing, usage, and customer lifecycle management
• Develop robust data transformation, cleansing, validation, and enrichment processes to improve customer data quality and consistency
• Design and develop Lakehouse architectures utilizing Delta Lake and Bronze, Silver, and Gold data layers
• Build and maintain batch and real-time streaming data pipelines using Apache Spark and Databricks technologies
• Develop and optimize Delta Lake tables, including partitioning, schema evolution, and performance tuning strategies
• Implement and manage data workflows using Databricks Workflows, Delta Live Tables, and related orchestration technologies
• Work with Databricks Unity Catalog to establish data governance, access controls, security, and metadata management
• Integrate Databricks with existing customer platforms, CRM systems, billing applications, enterprise applications, and downstream data consumers
• Optimize Spark jobs, SQL queries, and data processing workloads to improve platform performance, scalability, and cloud cost efficiency
• Develop reusable data engineering frameworks, libraries, and coding standards to improve development efficiency
• Implement CI/CD pipelines, version control, and Infrastructure-as-Code practices to support reliable platform deployments
• Collaborate with Data Architects, Analytics teams, and business stakeholders to translate customer platform requirements into scalable technical solutions
• Ensure customer data solutions meet enterprise security, privacy, regulatory, governance, and data quality requirements
• Troubleshoot complex data pipeline failures, performance bottlenecks, and integration challenges across enterprise environments
• Support production deployments, platform enhancements, and ongoing operational improvements
• Contribute to the development of a modern customer data foundation supporting advanced analytics, customer insights, personalization, and AI-driven initiatives

Requirements:

• 5+ years of experience in Data Engineering, Software Engineering, or a related technical discipline
• 2+ years of hands-on experience developing solutions using Databricks
• Strong proficiency in Python, PySpark, and SQL
• Extensive experience with Apache Spark and distributed data processing
• Proven experience designing, developing, and maintaining production-grade ETL/ELT data pipelines
• Strong understanding of Databricks Lakehouse architecture and Delta Lake
• Experience implementing data ingestion, transformation, and processing frameworks across complex enterprise environments
• Hands-on experience with Delta Lake, including Delta tables, schema evolution, and optimization techniques
• Experience working with cloud platforms, particularly Microsoft Azure, AWS, or Google Cloud Platform
• Strong understanding of data warehousing, data modelling, and modern data architecture principles
• Experience developing batch processing and, ideally, real-time streaming solutions
• Familiarity with data orchestration, workflow automation, and production pipeline monitoring
• Understanding of data governance, security, access controls, and data quality frameworks
• Experience with Git, CI/CD pipelines, and modern software development practices
• Experience integrating data from multiple enterprise applications and legacy systems
• Understanding of customer data structures, customer information management, and enterprise data integration
• Strong analytical and problem-solving skills, with the ability to troubleshoot complex engineering challenges
• Excellent communication skills and the ability to collaborate effectively with technical and non-technical stakeholders
• Strong expertise in Apache Spark, PySpark, Python, SQL, and Delta Lake
• experience developing production-grade data pipelines, integrating complex enterprise systems, and delivering scalable Lakehouse solutions within large-scale cloud environments
• Databricks Certified Data Engineer Associate or Professional certification
• Experience with Databricks Unity Catalog, Delta Live Tables, and Databricks Workflows
• Experience implementing Medallion Architecture and enterprise Lakehouse solutions
• Knowledge of Databricks Asset Bundles and automated deployment frameworks
• Experience with Databricks SQL and developing optimized analytical workloads
• Familiarity with MLflow, Feature Store, and machine learning operationalization
• Experience supporting Generative AI, Large Language Models (LLMs), and AI-driven data solutions
• Knowledge of cloud-native data services, including Azure Data Factory, AWS Glue, S3, ADLS, and related technologies
• Experience with Terraform, Infrastructure-as-Code, and cloud platform automation
• Familiarity with dbt, Airflow, Kafka, Fivetran, or other modern data engineering technologies
• Experience migrating legacy customer platforms, data warehouses, CRM environments, or traditional ETL frameworks to Databricks
• Experience integrating CRM, customer billing, customer relationship management, and enterprise operational systems
• Previous experience within the utilities, energy, power generation, transmission, distribution, or wider infrastructure sectors
• Understanding of utilities customer data domains, including customer accounts, service connections, metering, billing, energy consumption, and customer interactions
• Experience working within Fortune 500 organizations or other highly complex enterprise environments
• Previous utilities or energy sector experience
• Experience within utilities, energy, or other highly regulated infrastructure environments
• candidates have worked on large-scale customer data or platform transformation projects