Data Engineer / Platform Administrator (Azure Databricks)

Vanderbilt University Medical Center is an academic health system providing healthcare, education, research, and health information technology services. The Data Engineer / Platform Administrator will administer the Azure Databricks platform, build automation and monitoring solutions, and develop scalable data pipelines while supporting platform governance, performance, and data accuracy.

Responsibilities:

• Administer our Azure Databricks environment of 100+ workspaces, including compute policies, access controls, and governance, working with senior team members on larger architectural decisions
• Build automation and Infrastructure as Code to provision and manage workspaces at scale
• Build solutions for proactive monitoring of platform health, performance, and costs
• Test new Databricks features and promote best practices to teams across the organization
• Develop tools and services that other teams use to create, test, and deliver data-related assets
• Design and develop performance-focused data pipelines that support a variety of source systems (flat files, APIs, databases, and more)
• Troubleshoot issues with production pipelines and work with internal stakeholders to deploy fixes
• Build reusable solutions for recurring problems and help ensure data accuracy and integrity
• Work with stakeholders to identify data needs and look for opportunities to improve platform scalability

Requirements:

• Bachelor's degree
• 1+ year of experience with Databricks (certifications are a plus)
• 2+ years of Python in an object-oriented paradigm
• 2+ years implementing cloud solutions with Azure, AWS, or GCP
• 1+ year of SQL experience
• Has basic knowledge of the architecture of the systems which support more than one core business area. Demonstrates knowledge of how the core business areas' applications and support systems are structured, how various components and parts interact, as well as what protocols and interfaces are used for communications and cooperation between modules and components. Participates in the overall planning and design of the specific core business systems. Able to train and educate others in the structure and components of systems architecture for those core business areas
• Has basic knowledge of the architecture of the business systems which support more than one core business area. Demonstrates knowledge of how the core business areas' applications and support systems are structured, how various components and parts interact, as well as what protocols and interfaces are used for communications and cooperation between components
• Has basic knowledge of leveraging cloud-based architecture to support the life cycle of a data lake. Demonstrates an understanding of when and how to scale cloud components to provide the required level of performance and support of a data lake
• Demonstrates a basic understanding of the effective application, operation, or practice in complex data lakes. Experienced in all core features and functions. Demonstrates a thorough understanding of advanced concepts. Applies concepts and tools consistently in delivering data lakes. May have successfully completed training in advanced techniques, features, and tools
• Proficient understanding of fundamental data mart concepts, encompassing their purpose, significance within data warehousing, and the advantages they provide in supporting data analytics efforts
• Demonstrates a foundational understanding of data warehousing and its role within an organization. Possesses knowledge of basic corporate data warehouse models and their purposes. Understands the core components of data warehousing systems. Familiar with fundamental data warehouse structures both in theory and basic practical application. Capable of basic performance considerations for data warehousing components
• Experience with Infrastructure as Code solutions such as Terraform
• Experience with CI/CD automation in GitLab, GitHub, or Azure DevOps
• Experience with REST data services, APIs, and microservices
• Experience with containerization using Docker or Podman
• Experience with PowerShell and/or Bash
• A link to your GitHub profile or other personal code examples