Director, Search & AI Evaluation

Role Overview

• Define and drive the long-term strategy for search relevance, retrieval evaluation, ranking optimization, and AI system quality.
• Lead initiatives focused on improving: Search relevance and ranking quality.
• Semantic retrieval and vector search
• Retrieval-augmented generation (RAG)
• AI grounding and hallucination mitigation
• User discovery and engagement outcomes
• Establish scalable evaluation methodologies for search, retrieval, recommendation, and LLM-powered systems.
• Guide experimentation and optimization across lexical, semantic, hybrid, and AI-assisted retrieval architectures.
• Partner with Product and Engineering leadership to align search and AI investments with customer and business priorities.
• Influence technical direction for retrieval systems, evaluation infrastructure, and AI quality frameworks across platforms.
• Define and operationalize evaluation frameworks for search and generative AI systems, including: IR metrics (e.g., NDCG, recall, precision)
• LLM and RAG evaluation methodologies
• Grounding and faithfulness evaluation
• Human evaluation and annotation strategies
• Online experimentation and A/B testing
• Establish best practices for offline benchmarking, online experimentation, and reproducible evaluation workflows.
• Build scalable processes for benchmark creation, annotation quality, evaluation governance, and performance reporting.
• Drive rigorous, evidence-based decision-making across AI and search initiatives.
• Champion responsible AI practices focused on quality, reliability, trust, and measurable user impact.
• Lead, mentor, and grow a high-performing team of Data Scientists and Analysts.
• Create a culture of scientific rigor, accountability, collaboration, innovation, and continuous learning.
• Partner closely with Product Managers, Engineers, UX Researchers, and Applied AI teams to deliver impactful AI capabilities.
• Translate complex technical findings into clear business insights and strategic recommendations for senior stakeholders and executive leadership.

Requirements

• Master’s or PhD in Computer Science, Data Science, Machine Learning, Information Retrieval, Statistics, NLP, or a related quantitative field.
• 10+ years of experience in Data Science, Machine Learning, Information Retrieval, Search Relevance, Evaluation Systems, or Applied AI.
• Significant experience leading and scaling high-performing technical teams in complex, cross-functional organizations.
• Deep expertise in: Search relevance and ranking systems.
• Information retrieval and semantic search
• Retrieval-augmented generation (RAG)
• Evaluation methodologies for IR and generative AI systems
• Experimentation frameworks and A/B testing
• Strong experience with: Vector retrieval and hybrid search architectures.
• LLM evaluation and AI quality measurement
• Embeddings, reranking, and retrieval orchestration
• Evaluation datasets, benchmarking, and annotation workflows
• Advanced programming skills in Python.
• Experience with modern AI/ML frameworks and tooling (e.g., PyTorch, Hugging Face, LangChain, LangGraph, Haystack).
• Experience working with large-scale datasets, distributed data/ML platforms, and production AI systems.
• Strong understanding of statistical analysis, experimentation design, and evaluation science.
• Excellent communication and stakeholder management skills, including experience influencing senior leadership.
• Demonstrated ability to balance strategic leadership with pragmatic execution in rapidly evolving AI environments.

Tech Stack

• Python
• PyTorch

Benefits

• Comprehensive Pension Plan
• Home, office, or commuting allowance.
• Generous vacation entitlement and option for sabbatical leave
• Maternity, Paternity, Adoption, and Family Care leave
• Flexible working hours
• Personal Choice budget
• Internal communities and networks
• Various employee discounts
• Recruitment introduction reward
• Employee Assistance Program (global)