Databricks Data Engineer

E Source is a research, data/analytics, and technology focused professional services firm in the utility industry. They are seeking a Databricks Data Engineer to design, build, and maintain scalable data pipelines and systems, collaborating with various teams to deliver effective data solutions.

Responsibilities:

• Design, develop, and maintain data pipelines, infrastructure, and systems using AWS, Python, Databricks, Spark, and PostgreSQL
• Collaborate with cross-functional teams to translate business requirements into technical solutions
• Provide technical guidance and mentorship to junior data engineers
• Develop and implement data engineering standards and best practices
• Monitor and troubleshoot data pipelines to ensure data quality, integrity, and availability
• Research industry trends and evaluate new data technologies
• Build and maintain data lakes to support analytics and business intelligence
• Manage the software development life cycle and DevOps activities
• Partner with data scientists and analysts to deliver effective data solutions
• Develop data validation and data analysis tools

Requirements:

• Bachelor's degree in computer science, information technology, or a related field
• Five or more years of experience in data engineering or a related role
• Expert proficiency in Python, SQL, Databricks, and Spark
• Experience building cloud-based data pipelines in AWS
• Knowledge of data governance, security, privacy, and retention practices
• Strong communication, collaboration, and problem-solving skills
• Experience working in agile environments using tools such as Jira
• Proficiency with Git, Docker, and CI tools such as Jenkins
• Passion for data and continuous improvement
• Ability to work independently and proactively
• Experience optimizing ETL workflows in Databricks
• Knowledge of data lineage and data quality practices
• Experience supporting data science and machine learning solutions
• Prior experience in the energy or utility industry
• Master's degree in a relevant STEM field