Senior Azure Databricks Architect

 

Job Title: Senior Azure Databricks Architect    

Location: Dallas, TX

Work: Hybrid role – 3 days a week                                    

Interview: Three rounds - Video interview & In person

 

Job Description:

We are hiring a hands-on Technical Architect to lead enterprise-scale migrations to the Databricks Lakehouse for our US customers. You will lead the architecture and technical execution of a Palantir Foundry to Databricks migration. The candidate will be responsible for assessing the existing Palantir application, defining the target Databricks architecture, mapping existing capabilities to Databricks services, and providing technical leadership throughout the migration.

The solution involves migration of capabilities including data ingestion and transformation, AI-based information extraction from structured and unstructured documents, large-scale data comparison, dashboards and drill-down reporting, chatbot capabilities, and business workflows/applications.

The ideal candidate should have strong expertise in Databricks, Lakehouse architecture, data engineering, AI/ML workloads, and cloud-native data platforms, along with the ability to understand and decompose existing Palantir Foundry capabilities into equivalent or improved Databricks-based solutions. You will own the end-to-end technical solution: architecture, design, deployment, performance and every major technology decision. You will work directly with customer business and IT stakeholders, turn business requirements into clear technical requirements, and guide a delivery team of engineers. This is a builder role: you are expected to design, write and review code, not only draw diagrams

 

Roles & Responsibilities:

 

Architecture ownership

• Own the target-state Lakehouse architecture on Databricks: medallion layers, Delta Lake design, Unity Catalog structure, workspace topology and integration patterns.
• Define the migration architecture and strategy: source assessment, lift-and-shift vs re-engineer decisions, wave planning, coexistence and cutover approach.
• Produce and maintain architecture artefacts: solution blueprints, ADRs (architecture decision records), data flow and security designs.
• Identify the validation criteria for the business users and build validation framework
• Define the target-state Databricks architecture for the migrated application.
• Design scalable and secure Lakehouse architecture using Databricks.
• Define the appropriate architecture across Bronze, Silver and Gold layers.
• Design data ingestion, transformation, processing and serving patterns.
• Define standards for Delta Lake, Unity Catalog, compute, storage, workloads and environments.
• Design solutions for batch and near-real-time data processing.

Design and technology decisions

• Be the final technical authority on tool, framework and pattern choices across ingestion, transformation, orchestration, governance and ML.
• Evaluate trade-offs on cost, performance, scalability, security and maintainability, and document the reasoning.
• Set engineering standards: coding guidelines, reusable frameworks, data quality and testing approaches.
• Analyze the existing Palantir Foundry application, including its pipelines, datasets, applications, workflows and AI capabilities.
• Create a feature-by-feature mapping between existing Palantir capabilities and Databricks capabilities.
• Analyze the existing Palantir Foundry application, including its pipelines, datasets, applications, workflows and AI capabilities.
• Create a feature-by-feature mapping between existing Palantir capabilities and Databricks capabilities.
• Identify capabilities that can be migrated directly versus those requiring redesign.
• Define migration patterns for:
  • Data ingestion pipelines
  • Data transformation pipelines
  • Data quality and validation
  • Data comparison and reconciliation
  • AI/LLM-based information extraction
  • Dashboards and reporting
  • Chatbot capabilities
  • Business workflows and actions
  • Document processing
• Develop migration roadmaps, architecture decisions and technical dependencies.
• Identify opportunities to improve scalability, maintainability and performance during migration.

Deployment

• Design and implement CI/CD for Databricks using Databricks Asset Bundles, Git and Terraform.
• Plan and run production cutovers, parallel runs, data reconciliation and rollback procedures.
• Set up environment strategy (dev/test/prod), release management and operational runbooks.
• Design robust data pipelines using Lakeflow / Spark Declarative Pipelines, Apache Spark and Delta Lake.
• Design ingestion from databases, APIs, files, documents and other enterprise data sources.
• Establish data quality, validation, lineage and observability mechanisms.
• Design incremental processing and change-data-capture patterns where required.
• Optimize Delta tables, Spark workloads and Databricks compute for performance and cost.

Performance and cost

• Own performance of pipelines and queries: Spark tuning, partitioning and clustering (Liquid Clustering, Z-order), Photon, caching and file-size optimisation.
• Size clusters and SQL warehouses, and drive DBU cost monitoring and optimisation (FinOps).
• Define and track SLAs for data freshness, job run-times and query latency.

Business-to-technical translation

• Run workshops with business and IT stakeholders to understand processes, data needs and outcomes.
• Convert business requirements into technical requirements, data models, user stories and acceptance criteria.
• Explain technical options, risks and trade-offs clearly to non-technical and executive audiences.

 

AI / GenAI Architecture

• Architect solutions for extracting information from structured and unstructured documents.
• Design RAG and LLM-based solutions where applicable.
• Evaluate appropriate Databricks capabilities for:
  • Document processing
  • Information extraction
  • Embeddings and vector search
  • LLM inference
  • Model serving
  • AI agents / conversational applications
• Define approaches for migrating existing Palantir AIP-based extraction capabilities into Databricks.
• Ensure AI solutions are scalable, secure, explainable and production-ready.
• Design the target architecture for dashboards, drill-down reporting and analytical applications.
• Define semantic/data serving layers required by business applications.
• Architect chatbot and conversational analytics solutions.
• Design integration between Databricks and downstream applications/APIs.

Technical leadership and mentoring

• Guide and mentor data engineers and developers through design reviews, code reviews and pairing.
• Lead a team of data engineers, ML/AI engineers and developers.
• Conduct architecture reviews and technical design sessions.
• Create HLD/LLD documents, architecture diagrams and migration plans.
• Establish Databricks development standards and best practices.
• Review code and implementation approaches.
• Work closely with business stakeholders, Palantir SMEs, cloud architects and application teams.
• Troubleshoot complex architectural and performance issues.
• Unblock the team on complex technical issues and lead root-cause analysis on production problems.
• Build team capability on Databricks through knowledge sessions and support for certifications.
• Work closely with business stakeholders, SMEs, cloud architects and application teams.

 

Qualifications & Requirements:

• 12+ years in data engineering, data warehousing or data architecture, with 5+ years hands-on on Databricks.
• Bachelor's or Master's degree in Computer Science, Engineering, Information Technology or related discipline.
• 5+ years of experience working with modern cloud data platforms.
• Proven experience designing enterprise-scale data platforms.
• Experience leading technical teams and architecture initiatives.
• Strong communication and stakeholder-management skills.
• Ability to translate business requirements into scalable technical architectures.
• Has led at least one large-scale enterprise migration to Databricks end to end (multi-TB to PB data, hundreds of pipelines, phased cutover).
• Deep hands-on skill in Apache Spark (PySpark, Spark SQL), SQL and Python.
• Strong knowledge of Delta Lake, Unity Catalog, Databricks Workflows, Delta Live Tables and Databricks SQL.
• Proven experience migrating from legacy platforms such as Teradata, Oracle, Netezza, SQL Server, Hadoop/Cloudera or Informatica.
• Track record of owning architecture and technology decisions on enterprise programs.
• Experience turning business requirements into technical designs and working directly with US customer stakeholders.
• Experience mentoring and technically leading engineering teams, including onsite–offshore models.
• Excellent communication and presentation skills.

 

Technology Skills

Area

Skills expected

Databricks platform

Lakehouse architecture, Delta Lake, Unity Catalog, Databricks SQL and SQL Warehouses, Workflows/Jobs, Delta Live Tables, Photon, Serverless compute

Migration sources

Teradata, Oracle, Netezza, SQL Server, Hadoop/Cloudera (Hive, Impala, Spark), Informatica, DataStage, SSIS, SAS

Migration tooling

Lakebridge / BladeBridge or similar code-conversion tools, automated data validation and reconciliation frameworks

Data engineering

PySpark, Spark SQL, Python, Structured Streaming, Auto Loader, Kafka / Event Hubs / Kinesis, medallion architecture, dbt (plus)

Data modelling

Dimensional modelling, Data Vault, Lakehouse modelling patterns, SCD handling

Performance

Spark UI analysis, AQE, partitioning, Liquid Clustering / Z-order, OPTIMIZE / VACUUM, cluster and warehouse sizing

Governance and security

Unity Catalog permissions, row/column-level security, lineage, data quality (DLT expectations, Great Expectations), PII handling

AI/ML on Databricks

MLflow, Feature Store, Model Serving, Mosaic AI, Vector Search, GenAI use cases

Cloud

Databricks on AWS, Azure or Google Cloud Platform: object storage, networking (VNet/VPC, Private Link), IAM

DevOps and FinOps

Databricks Asset Bundles, Git, Azure DevOps / GitHub Actions / Jenkins, Terraform, DBU cost monitoring