Databricks Platform & Data Engineer

Title: Databricks Platform & Data Engineer

Location: Plano, TX & Minneapolis, MN (Hybrid)

Type: Contract

Role Summary:

• Client is seeking a senior Databricks Platform & Data Engineer to join our Databricks Center of Excellence (CoE). This role blends platform engineering, architecture, and hands-on data engineering, with a focus on building and scaling enterprise-grade Databricks environments for large, regulated clients.
• The ideal candidate will lead platform setup, governance, and adoption, while partnering directly with clients to design and deliver high-impact use cases leveraging the latest Databricks capabilities (e.g., Dataflow, Unity Catalog, AI/BI, Genie).
• This is a client-facing consulting role requiring strong technical depth, communication skills, and the ability to translate business requirements into scalable data platform solutions.

Key Responsibilities:

Platform Engineering & Architecture

• Design and deploy enterprise-scale Databricks Lakehouse platforms across AWS/Azure
• Establish secure, governed environments using Unity Catalog, role-based access controls, and data lineage
• Define platform standards, reusable patterns, and guardrails for scalable adoption
• Optimize platform performance, cost, and reliability for production workloads
• Implement CI/CD, environment promotion, and DevOps automation for Databricks

Databricks Feature Enablement

• Lead adoption of modern Databricks capabilities including:
• Unity Catalog: Centralized governance, access control, lineage
• Dataflow / declarative pipelines: build scalable ingestion and transformation frameworks
• Genie / AI-assisted development: accelerate developer productivity and data accessibility
• Enable AI/BI dashboards, model serving, and advanced analytics use cases

Data Engineering & Use Case Delivery

• Build and optimize batch and streaming data pipelines using Spark and Delta Lake
• Develop data products and domain-oriented pipelines aligned to enterprise data strategies
• Lead end-to-end use case delivery, from requirements to production deployment
• Drive data quality, observability, and pipeline reliability

Client Engagement & Advisory

• Act as a trusted advisor to client stakeholders (architecture, data, risk, and business teams)
• Translate business requirements into technical architecture and delivery roadmaps
• Lead workshops, solution design sessions, and platform adoption strategies
• Support proposals, solutioning, and client innovations within regulated industries

CoE Contribution:

• Build and contribute to Client accelerators, frameworks, and reusable assets
• Define best practices, reference architectures, and playbooks for enterprise Databricks adoption
• Mentor junior engineers and support capability building across the CoE

Required Qualifications:

• 12+ years of experience in data engineering, platform engineering, or data architecture
• 5+ years hands-on experience with Databricks in large enterprise environments

Deep expertise in: -

• Apache Spark (Scala/Python)
• Delta Lake and Lakehouse architectures
• Databricks workspace setup, cluster policies, and job orchestration
• Strong experience with cloud platforms (AWS, Azure, or Google Cloud Platform)
• Experience implementing data governance, security, and compliance controls
• Proven ability to design and deliver scalable, production-grade data platforms
• Strong client-facing and communication skills

Preferred Qualifications:

• Experience in financial services or other regulated industries
• Familiarity with data governance frameworks, regulatory reporting, and risk data environments
• Databricks certifications (e.g., Data Engineer, Solutions Architect) Experience with:
• Real-time streaming (Kafka, Structured Streaming)
• MLOps / Model Serving
• Data marketplace / data product architectures
• Exposure to AI-assisted development workflows and agent-based tooling