Senior Data Platform Architect Databricks (PySpark)

Job Title: Senior Data Platform Architect - Databricks (PySpark)

Location: Dallas, TX - Onsite

Job Summary

We are looking for an experienced Senior Data Platform Architect - Databricks (PySpark) to design and drive enterprise-scale data platform architecture and modernization initiatives. The role requires strong expertise in Data Engineering, Databricks Lakehouse, Python, PySpark, Spark SQL, distributed data processing, and cloud-native data architectures.

The selected candidate will collaborate closely with the Enterprise Architecture team to define target-state architectures, establish technology roadmaps, and develop scalable, secure, and reusable data architecture solutions. The role will also provide technical leadership to engineering teams and support AI/ML and Generative AI initiatives through the development of AI-ready data platforms.

Key Responsibilities

• Design and define scalable enterprise data platform architectures using Databricks Lakehouse and modern cloud-native technologies.
• Collaborate with Enterprise Architecture teams to establish target-state architectures, technology roadmaps, and data modernization strategies.
• Create, maintain, and govern reference architectures, solution blueprints, design patterns, and architecture standards.
• Design and implement scalable batch, streaming, and real-time data processing solutions.
• Provide technical direction for distributed data processing using Python, PySpark, Spark SQL, and the Hadoop ecosystem.
• Drive the architecture and evolution of enterprise data platforms across Azure, AWS, or Google Cloud Platform cloud environments.
• Define architecture standards and best practices for data ingestion, transformation, storage, processing, and integration.
• Support AI/ML and Generative AI initiatives by designing AI-ready data platforms and enabling scalable, high-quality data pipelines.
• Conduct architecture reviews and ensure solutions meet enterprise requirements for security, scalability, performance, governance, and reliability.
• Evaluate emerging data technologies and recommend suitable solutions aligned with enterprise architecture standards.
• Mentor data engineering teams and provide technical leadership throughout solution design, development, and implementation.
• Collaborate with business stakeholders, technical teams, and architecture leaders to influence technology decisions and ensure alignment with business objectives.

Required Technical Skills & Experience

• 7-12 years of experience in Data Engineering, Data Platforms, or Solution Architecture.
• Strong hands-on experience with Python, PySpark, and Spark SQL.
• Strong understanding of Apache Spark and distributed data processing.
• Experience working with the Hadoop ecosystem, including technologies such as HDFS, Hive, and related big data platforms.
• Hands-on expertise in Databricks Lakehouse architecture and implementation.
• Experience designing and implementing modern cloud-native data architectures.
• Experience with at least one major cloud platform: Microsoft Azure, AWS, or Google Cloud Platform (Google Cloud Platform).
• Strong knowledge of batch, streaming, and real-time data processing architectures.
• Experience defining enterprise data architecture standards, reference architectures, solution blueprints, and reusable design patterns.
• Understanding of data security, governance, scalability, performance optimization, and architecture best practices.
• Strong communication, stakeholder management, and technical leadership skills.

Preferred Qualifications

• Databricks certification.
• Microsoft Azure, AWS, or Google Cloud Platform cloud certification.
• Previous experience working with Citibank consumer banking or wealth management platforms.
• Exposure to Enterprise Architecture frameworks, preferably TOGAF.
• Experience in Banking, Financial Services, or large-scale enterprise data modernization programs.
• Exposure to AI/ML and Generative AI data platform architecture.

Key Skills

• Data Platform Architecture
• Databricks Lakehouse
• Python
• PySpark
• Apache Spark / Spark SQL
• Hadoop Ecosystem
• Distributed Data Processing
• Cloud Data Architecture - Azure / AWS / Google Cloud Platform
• Batch, Streaming & Real-Time Data Processing
• Enterprise Architecture
• Reference Architecture & Solution Blueprints
• Data Governance & Security
• AI/ML & Generative AI-Ready Data Platforms
• Technical Leadership & Stakeholder Management