Databricks Architect

Role : Databricks Architect

Location : Tarrytown New York (Onsite)

• Evaluate and drive adoption of new Databricks capabilities, including Lakebase, Databricks Apps, Lakehouse Federation, Serverless, Genie and AI/ML capabilities.
• Conduct architecture reviews, define reusable patterns and provide technical governance across projects.
• Partner with business, engineering, and architecture teams to define target-state architectures and modernization roadmaps.

Key Responsibilities & Skills

• Lead enterprise-scale data platform modernization initiatives, driving migration from AWS EMR, Apache NiFi, and legacy ETL frameworks to the Databricks Lakehouse Platform.
• Architect and implement scalable Lakehouse solutions using Databricks, Delta Lake, Unity Catalog, and Databricks Workflows.
• Design and govern end-to-end data pipelines for batch, streaming, CDC, and real-time data integration workloads.
• Provide architecture leadership for large-scale AWS-based data ecosystems leveraging S3, IAM, Redshift, Glue Catalog, Airflow, and Databricks.
• Develop enterprise data architecture standards covering data modelling, metadata management, lineage, governance, security, and compliance.
• Drive adoption of Databricks best practices including Delta Live Tables (DLT), Auto Loader, Unity Catalog, Serverless Compute, Lakehouse Federation, and advanced optimization techniques.
• Lead replatforming and migration programmes involving PySpark applications, Airflow DAGs, EMR workloads, Redshift integrations, and NiFi pipelines.
• Partner closely with business stakeholders, enterprise architects, product owners, and data science teams to define technology roadmaps and target architectures.
• Establish governance frameworks using Unity Catalog, data quality controls, observability, monitoring, and operational excellence practices
• Provide technical leadership, mentoring, architecture reviews, design governance, and solution sign-offs across multiple delivery teams.
• Lead architecture workshops, executive presentations, solution assessments, and technology evaluations.

Mandatory Technical Skills

• Databricks Lakehouse Platform
• Delta Lake, Delta Live Tables (DLT)
• Unity Catalog
• Databricks Workflows
• Auto Loader
• PySpark, Spark SQL, Python
• AWS (S3, IAM, Glue, Redshift, EMR, Lambda)
• Apache Airflow
• CDC & Data Migration Frameworks
• Data Governance & Security
• CI/CD, GitHub, Jenkins

JD:

• Lead enterprise-scale Databricks Lakehouse Architecture design and implementation on AWS.
• Drive large-scale data platform modernisation and cloud transformation initiatives.
• Architect scalable Medallion Architecture (Bronze, Silver, Gold) data platforms.
• Lead migration of legacy EMR, NiFi, Redshift, and ETL workloads to Databricks.
• Design high-performance batch and real-time data processing solutions.
• Build robust ingestion frameworks using Auto Loader, Delta Lake, and Structured Streaming.
• Define enterprise data governance and security standards using Unity Catalog.
• Architect metadata-driven and reusable PySpark-based ETL/ELT frameworks.
• Establish best practices for performance tuning, scalability, and cost optimisation.
• Design and implement data quality, lineage, and observability frameworks.
• Drive adoption of CI/CD, DevOps, Infrastructure as Code, and automation practices.
• Collaborate with business, analytics, and engineering teams to define target-state architectures.
• Conduct architecture reviews and provide technical leadership across multiple projects.
• Mentor architects and senior engineers on Databricks and AWS best practices.
• Design secure and scalable solutions leveraging AWS services (S3, Glue, Athena, Lambda, IAM, Redshift).
• Implement and govern enterprise-wide data access, compliance, and security controls.
• Evaluate and adopt latest Databricks capabilities such as DLT, Lakehouse Federation, Serverless, and MLflow.
• Enable AI/ML, Generative AI, and advanced analytics use cases on the Lakehouse platform.
• Create architecture roadmaps, migration strategies, standards, and governance frameworks.
• Act as the primary technical advisor for customer leadership on data strategy and platform evolution