Databricks Technical Lead/Architect

Hi, 

Hope you’re doing well!! 

Please find the requirement below. If you find yourself comfortable with the requirement please reply back with your updated resume or call me back at   

Position: Databricks Technical Lead/Architect
Location: Remote
Duration: 6+ Months
Visa: Client won't be able to sponsor any visa
LinkedIN ID

MUST BE ABLE TO TAKE A GLIDER

Key Responsibilities

·                     Technical Leadership: Establish architectural patterns and best practices for medallion architecture, Delta Lake optimization, data mesh principles, pipeline orchestration, and cross-domain data sharing. Lead technical code reviews for data engineering teams, providing detailed feedback on PySpark, SQL, and architectural patterns; ensure code quality, performance optimization, and adherence to best practices.

·                     Architecture & Strategy: Define the overall Databricks platform architecture, including workspace design, Unity Catalog governance model, storage patterns, compute strategies, and integration with broader enterprise data ecosystems.

·                     DevOps & Automation: Design CI/CD architecture for Databricks assets, including Git-based development workflows, automated testing frameworks, deployment pipelines, and infrastructure-as-code patterns using Terraform or similar tools.

·                     Governance & Security: Design and implement comprehensive governance frameworks within Unity Catalog, including multi-catalog strategies, metastore configuration, identity federation, fine-grained access controls, attribute-based access control (ABAC), data classification, and compliance requirements.

·                     Platform Design: Architect solutions for compute resource allocation (all-purpose clusters, job clusters, SQL warehouses, serverless), autoscaling strategies, cost optimization, and workload isolation across teams and environments.

·                     Data Modeling & Standards: Define enterprise data modeling standards, schema design patterns, slowly changing dimensions (SCD) strategies, CDC architectures, and data quality frameworks that scale across multiple domains.

·                     Performance & Optimization: Architect solutions for query optimization, storage layout strategies (partitioning, Z-ordering, liquid clustering), caching, materialized views, and monitoring/observability across the platform.

·                     Integration Architecture: Design integration patterns with upstream source systems (APIs, databases, streaming), downstream analytics tools (Power BI, Tableau), and adjacent cloud services (Azure Synapse, AWS services, etc.).

·                     Collaboration & Enablement: Partner with engineering teams, data teams, and stakeholders to translate business requirements into architectural blueprints; mentor engineers on implementation of architectural patterns.

·                     Innovation & Evaluation: Stay current with Databricks platform evolution, evaluate new features (Lakehouse Federation, AI/ML capabilities, streaming enhancements), and drive adoption of capabilities that deliver business value.

Required Skills & Experience

·                     Current, hands-on development experience is essential. This role requires the ability to read, write, debug, and lead code reviews of production PySpark and SQL code—not just architectural oversight. Candidates must demonstrate recent (within past 12 months) hands-on development work including debugging live code, explaining business logic and technical implementation, optimizing queries, and implementing data pipelines. Ability to comprehend and explain unfamiliar code samples is a core requirement.

·                     Advanced proficiency in SQL and Python/PySpark with demonstrated ability to write complex transformations, optimize query performance, explain code logic at both business and technical levels, and troubleshoot production issues. Must be comfortable with CTEs, window functions, table-valued functions, APPLY operators, advanced SQL operators, and PySpark DataFrame/RDD operations. Current, active coding skills required—not aspirational or theoretical knowledge.

·                     7+ years of experience in data architecture, data engineering, or platform engineering roles, with at least 3+ years focused on Databricks platform architecture.

·                     Expert-level knowledge of Databricks platform components: Unity Catalog, Delta Lake, Delta Live Tables, Workflows, SQL Warehouses, MLflow, and Databricks SQL.

·                     Deep expertise in Unity Catalog governance, including metastore design, catalog/schema strategies, permission models, data lineage, and multi-workspace/multi-cloud patterns.

·                     Strong architectural background in cloud platforms (Azure, AWS, or Google Cloud Platform), including storage services, identity management (Azure AD, AWS IAM), networking, and security best practices.

·                     Proven experience designing enterprise-scale data architectures, including medallion/multi-hop architectures, data mesh patterns, domain-driven design, and data product frameworks.

·                     Hands-on experience with infrastructure-as-code (Terraform, ARM templates, CloudFormation) for platform configuration and governance automation.

·                     Strong understanding of DevOps practices, CI/CD pipelines, version control strategies, and automated testing for data platforms.

·                     Experience with performance tuning, cost optimization, and capacity planning for large-scale data platforms.

Preferred Qualifications

·                     Databricks certification (e.g., Databricks Certified Data Engineer Professional, Solutions Architect).  Note: Certification alone is insufficient; candidates must demonstrate current hands-on technical execution aligned with certification level.

·                     Cloud certifications such as Azure Solutions Architect, AWS Solutions Architect, or Google Cloud Platform Professional Data Engineer.

• Experience designing multi-tenant architectures with secure data isolation, cross-tenant data sharing, and compliance controls.
• Background in streaming architectures using Structured Streaming, Kafka, Event Hubs, or Kinesis.
• Exposure to machine learning operations (MLOps), feature stores, model serving, and AI/ML platform architecture.
• Experience with data mesh implementations, federated governance, and distributed data ownership models.
• Knowledge of analytics platforms (Power BI, Tableau, Looker) and their integration patterns with Databricks.
• Familiarity with domain-specific data models in education (CEDS), healthcare, finance, or operational domains.
• Experience with real-time CDC patterns, change data capture tools (Debezium, Qlik, Fivetran), and event-driven architectures.

Leadership & Soft Skills

·                     Professional presence and composure under pressure, including ability to handle unexpected technical challenges, maintain calm in production incidents, and communicate effectively during high-stress situations.

·                     Strategic thinking with ability to balance long-term architectural vision with pragmatic, incremental delivery.

·                     Exceptional communication skills to articulate complex architectural concepts to technical and non-technical stakeholders, including executive leadership.

·                     Proven ability to influence and drive consensus across multiple teams and organizational levels.

·                     Mentorship and enablement mindset to uplift engineering teams through knowledge sharing, documentation, and hands-on guidance.

·                     Strong problem-solving capabilities with a focus on root cause analysis and sustainable solutions.

·                     Commitment to quality, including comprehensive documentation, architectural decision records (ADRs), and knowledge transfer.

Position Summary

• This role is ideal for a seasoned architect who thrives on designing elegant, scalable solutions and wants to shape the foundation of a next-generation Lakehouse platform. The Databricks Architect will serve as the technical authority for the platform, driving architectural excellence, governance maturity, and innovation that enables the organization to unlock the full value of its data assets across analytics, data science, and operational use cases.

Thanks & Regards:

Amar Pratap

Senior Technical Recruiter
VISION INFOTECH INC
Phone: ext 531
Direct:  
Email:  
368 Main Street, st #3, Melrose MA 02176

E-Verified Company