Apetan Consulting LLC is seeking a highly skilled Senior Databricks Architect/Engineer to lead the design, development, and implementation of a next-generation data platform. This role will be instrumental in building a scalable Databricks-based solution to support treasury functions such as cash management, liquidity, forecasting, and payments, while enabling future BI and AI capabilities.
Responsibilities:
- Design and architect a scalable Databricks-based data platform from the ground up
- Perform hands-on development using Databricks, PySpark, and SQL
- Build and optimize data pipelines to ingest data from Azure/SQL Server environments
- Implement data architecture frameworks (e.g., medallion architecture)
- Configure and optimize Databricks clusters for performance and cost efficiency
- Collaborate with stakeholders to gather requirements and translate them into technical solutions
- Provide technical leadership and guidance to development teams
- Ensure platform stability through monitoring, optimization, and best practices
- Support integration with downstream tools for BI and AI use cases
Requirements:
- Strong hands-on experience with Databricks (Azure preferred)
- Proficiency in PySpark, SQL, and data engineering concepts
- Experience designing and implementing data platforms and architectures
- Solid understanding of Azure services (ADF, ADLS, SQL Server, etc.)
- Experience with data pipeline frameworks and ETL/ELT processes
- Knowledge of cluster management, performance tuning, and optimization
- Strong communication and stakeholder management skills
- Ability to work independently in a remote environment
- Experience in financial or treasury data domains (cash, liquidity, payments)
- Familiarity with Delta Lake, Unity Catalog, and data governance practices
- Exposure to BI tools and AI/ML integration