Product Data Scientist – AI Evaluation, Quality

Role Overview

• Own and extend our offline eval suite across products — datasets (capability + regression), judges, metrics
• Build and maintain online quality dashboards: resolution rate, CSAT, thumbs up/down, LLM-as-judge signals, error rate, latency
• Close the production feedback loop: mine failure patterns from real traffic → turn them into regression cases → propose fixes to Product and domain experts
• Harden methodology: judge stability, non-determinism handling
• Translate numbers into decisions – weekly syncs, clear trade-offs, no dashboards for their own sake

Requirements

• Python and SQL — you can build an analysis end-to-end
• Solid foundation in statistics — sampling, hypothesis testing, variance, understanding what a noisy metric is
• Analytical mindset — you start from the business question, not from the tool
• 3+ years in analyst / data scientist roles, at least one in a product context

Tech Stack

• Python
• SQL

Benefits

• Flexibility of traveling and working remotely or in a hybrid model across Europe.
• Stock Options Program available to every team member.
• Constant support and care to ensure your Finom experience is successful and fulfilling.
• Exclusive Work & Swim Program with corporate accommodation in Cyprus.