Design and build robust model evaluation frameworks, automating repetitive processes and maintaining a balanced approach to efficiency and depth in obtaining evaluation metrics and feedback.
Manage resource allocation and timelines, adjusting direction flexibly based on real-time information across all data streams in your product vertical.
Enhance dataset and process quality through seamless collaboration with vendors and outsourcing partners.
Establish labeling guidelines, monitor data quality, and improve tools and infrastructure to build a sustainable data operations framework.
Partner with Engineering and AI Model teams to align on top priority data needs, design tools such as analytical reports and dashboards, and clearly communicate project progress.

5+ years of experience working in an AI focused data operations organization.
A proven track record designing and executing large scale data or evaluation projects, including gathering, labeling, and post-processing data.
The ability to analyze messy and complex data, identify overarching patterns, and distill your findings into crisp annotation guidelines or model quality reports.
Proficiency with Python, LLMs, or other popular industry tools for automation.
Excellent communication and project management skills, and the ability to support several projects simultaneously.
A foundational understanding of and interest in LLMs/VLMs and multimodal AI.
Conviction that data is the key ingredient for the performance and assessment of AI models.

An open and inclusive culture and work environment.
Work closely with a collaborative, mission-driven team on cutting-edge AI technology.
Full health, dental, and vision benefits.
Flexible PTO and parental leave policy. Office closed the week of Christmas and New Years.

Model Evaluation, Data Quality Lead

Key skills