Senior Databricks Architect/Engineer / Remote

Apetan Consulting LLC is seeking a highly skilled Senior Databricks Architect/Engineer to lead the design, development, and implementation of a next-generation data platform. This role will be instrumental in building a scalable Databricks-based solution to support treasury functions such as cash management, liquidity, forecasting, and payments, while enabling future BI and AI capabilities.

Responsibilities:

• Design and architect a scalable Databricks-based data platform from the ground up
• Perform hands-on development using Databricks, PySpark, and SQL
• Build and optimize data pipelines to ingest data from Azure/SQL Server environments
• Implement data architecture frameworks (e.g., medallion architecture)
• Configure and optimize Databricks clusters for performance and cost efficiency
• Collaborate with stakeholders to gather requirements and translate them into technical solutions
• Provide technical leadership and guidance to development teams
• Ensure platform stability through monitoring, optimization, and best practices
• Support integration with downstream tools for BI and AI use cases

Requirements:

• Strong hands-on experience with Databricks (Azure preferred)
• Proficiency in PySpark, SQL, and data engineering concepts
• Experience designing and implementing data platforms and architectures
• Solid understanding of Azure services (ADF, ADLS, SQL Server, etc.)
• Experience with data pipeline frameworks and ETL/ELT processes
• Knowledge of cluster management, performance tuning, and optimization
• Strong communication and stakeholder management skills
• Ability to work independently in a remote environment
• Experience in financial or treasury data domains (cash, liquidity, payments)
• Familiarity with Delta Lake, Unity Catalog, and data governance practices
• Exposure to BI tools and AI/ML integration