Databricks Data Engineer with a Solution Architect

Diverse Lynx is seeking a highly skilled Databricks Data Engineer with a Solution Architect background to lead the design of enterprise-scale data solutions. The role involves establishing engineering standards, optimizing platform performance, and providing technical leadership for cloud modernization and data governance initiatives.

Responsibilities:

• Applies advanced expertise in Databricks, Spark, Delta Lake, and cloud-native architectures to design and govern large-scale enterprise data platforms
• Leads architecture decisions for cloud-based data and analytics solutions, ensuring scalability, security, resiliency, and operational excellence across Azure, AWS, or GCP environments
• Establishes and evolves platform standards, best practices, and governance frameworks for data engineering, streaming, AI/ML, and analytics workloads
• Recommends innovative technologies, architectural patterns, and optimization strategies to improve platform performance, reliability, and cost efficiency
• Works on and may lead highly complex, cross-functional initiatives involving cloud modernization, data platform transformation, and enterprise architecture adoption
• Designs and optimizes high-volume batch and real-time data pipelines, leveraging deep knowledge of Spark internals, Structured Streaming, and distributed computing principles
• Provides architectural leadership for Delta Lake, Unity Catalog, security, data governance, and data-sharing capabilities across multiple business domains
• Drives Infrastructure as Code (IaC), DevOps, and CI/CD adoption using Terraform, Databricks Asset Bundles (DABs), and modern software engineering practices
• Independently determines architectural approaches, technology choices, and implementation strategies for new and evolving platform capabilities
• Receives guidance on strategic business objectives while exercising significant discretion in technical and architectural decision-making
• Provides expert consultation to leadership, engineering teams, and business stakeholders, delivering solutions that may establish enterprise-wide standards and precedents
• Acts as a technical mentor and thought leader, elevating engineering capabilities and promoting architecture best practices across the organization
• Influences long-term cloud, data, and AI platform strategy, ensuring solutions align with business goals, regulatory requirements, and future growth needs

Requirements:

• Strong expertise in Spark, Delta Lake, Unity Catalog, cloud architecture, Infrastructure as Code (IaC), and modern data platform engineering principles
• Advanced expertise in Databricks, Spark, Delta Lake, and cloud-native architectures
• Experience in designing and governing large-scale enterprise data platforms
• Ability to lead architecture decisions for cloud-based data and analytics solutions
• Knowledge of scalability, security, resiliency, and operational excellence across Azure, AWS, or GCP environments
• Experience in establishing and evolving platform standards, best practices, and governance frameworks for data engineering, streaming, AI/ML, and analytics workloads
• Ability to recommend innovative technologies, architectural patterns, and optimization strategies
• Experience in designing and optimizing high-volume batch and real-time data pipelines
• Deep knowledge of Spark internals, Structured Streaming, and distributed computing principles
• Architectural leadership for Delta Lake, Unity Catalog, security, data governance, and data-sharing capabilities
• Experience in driving Infrastructure as Code (IaC), DevOps, and CI/CD adoption using Terraform and Databricks Asset Bundles (DABs)
• Ability to independently determine architectural approaches, technology choices, and implementation strategies
• Ability to provide expert consultation to leadership, engineering teams, and business stakeholders
• Experience in acting as a technical mentor and thought leader
• Ability to influence long-term cloud, data, and AI platform strategy