Own and extend our offline eval suite across products — datasets (capability + regression), judges, metrics
Build and maintain online quality dashboards: resolution rate, CSAT, thumbs up/down, LLM-as-judge signals, error rate, latency
Close the production feedback loop: mine failure patterns from real traffic → turn them into regression cases → propose fixes to Product and domain experts
Harden methodology: judge stability, non-determinism handling
Translate numbers into decisions – weekly syncs, clear trade-offs, no dashboards for their own sake

Python and SQL — you can build an analysis end-to-end
Solid foundation in statistics — sampling, hypothesis testing, variance, understanding what a noisy metric is
Analytical mindset — you start from the business question, not from the tool
3+ years in analyst / data scientist roles, at least one in a product context
Experience in quality analytics for ML systems — ranking, recommendations, classification, etc.
Hands-on experience evaluating LLM applications (RAG, agents, tool use, judges)
Experience building LLM agents — side projects, toy builds, personal experiments all count
AI-assisted coding is our default authoring environment
Fluent with AI coding or genuinely excited to become fluent fast
We care about what you ship and how clearly you think

Make a genuine impact on the product
Join our upward trajectory
Continuous personal and professional development
Empowering you to make a genuine impact on our evolving product
Flexibility of traveling and working remotely or in a hybrid model across Europe
Become a stock options holder
Stock Options Program available to every team member
Constant support and care to ensure your Finom experience is successful and fulfilling
Work & Swim Program in Cyprus for work-life balance

Product Data Scientist – AI Evaluation, Quality

Key skills