Senior ML Data Platform Developer

LawZero is a non-profit organization committed to advancing research and creating technical solutions for safe-by-design AI systems. They are seeking a Senior ML Data Platform Developer to architect and maintain a data engine for their next-generation models, ensuring seamless integration and efficient processing of large datasets.

Responsibilities:

• Design and maintain a layered storage architecture and partner with the Research team to ensure seamless integration with the training pipelines
• Scale and automate the data processing stack to handle petabytes of data and ensure its smooth operation
• Ensure efficient use of compute resources, including GPU access for compute-intensive data processing tasks
• Assist the Infrastructure team in provisioning the compute and storage environments to support scaling
• Ensure all datasets, including the intermediate outputs of each transformation stage, are versioned, reproducible, and fully traceable to meet specific and dynamic experiment needs, and are accompanied by datasheets, in accordance with internal Data Governance policies
• Collaborate with the Research team and other teams to understand their self-service needs around dataset exploration, sampling, and analysis, and develop proper tooling