BMLL transforms financial exchanges’ raw market data into an accessible and normalised view for customers across multiple use-cases.
Hold over 10 years of full-depth market data across approximately 100 venues, consisting of about 1.5 billion HDF5 files (~1.5 PB) stored on S3 and catalogued in Postgres.
Migrate processed layer to Delta Lake for both data and catalogue.
Migrate Data Products from Snowflake to Iceberg compatible Delta Tables.
Implement a Delta Lake architecture for large scale.
Model and partition L2/L3 order book data in Delta lake.
Implement metadata, compaction, and versioning.
Design a system that supports multiple delivery models downstream.
Implement a viable backup strategy at this scale with a 1 day RTO.

Industry experience with Databricks
Delta Lake
Unity Catalog
Delta Tables
Delta UniForm (Universal Format)
Industry experience with Apache iceberg.
AWS, S3 Tables, Lake formation.
Industry experience in developing on a Linux platform.
Experience with industry-standard development methodologies such as source code control, unit testing and continuous integration
A self starter with the ability to self-organise.
Strong problem-solving skills
Strong communication skills
Industry experience with Snowflake,
Industry experience with petabyte scale data volumes.
Industry experience with Python.
Experience working with financial data.

Databricks Lead Engineer – Contract

Key skills