Senior Data Scientist III - AI Evaluation and Prompt Engineering

LexisNexis is a global provider of information-based analytics and decision tools for professional and business customers. They are seeking a Senior Data Scientist who will focus on AI Evaluation and Prompt Engineering to enhance their AI-powered legal research tools by evaluating and improving large language model systems.

Responsibilities:

• Evaluate and tune LLM-powered features, such as prompt optimization, retrieval-augmented generation (RAG) systems, and semantic search performance
• Design and execute experiments to measure model quality, reliability, and user impact — translating technical findings into product recommendations
• Develop and maintain data pipelines for evaluating, tracking, and improving system performance (e.g., accuracy, latency, cost, and relevance metrics)
• Analyze structured and unstructured datasets (e.g., product usage logs, document metadata, LLM outputs) to identify patterns, insights, and areas for optimization
• Collaborate with product managers to translate product goals into measurable data science questions, propose next steps, and inform roadmap priorities
• Provide technical guidance to data engineers who build and maintain analytics and model evaluation infrastructure
• Communicate results clearly — through written reports, dashboards, and presentations — to technical and non-technical stakeholders
• Stay current on emerging practices in applied NLP, LLM evaluation, and data-driven product development, and thoughtfully adapt them to our environment

Requirements:

• 3–6 years of experience in data science, applied NLP, or AI product analytics, preferably within a SaaS or research-heavy product environment
• Strong proficiency in Python and data analysis libraries such as Pandas; solid working knowledge of SQL
• Ability to design and evaluate LLM-based systems (e.g., RAG pipelines, prompt evaluations, output scoring), even if not specialized in deep learning
• Experience with data exploration, experimentation, and reporting — from defining metrics to visualizing and interpreting results
• Comfort working with document-based datasets (e.g., text corpora, metadata, embeddings) and understanding information retrieval / semantic search concepts
• Excellent written and verbal communication skills — able to present complex ideas simply and persuasively across distributed teams
• Proven ability to self-direct, learn new tools and concepts quickly, and apply them pragmatically
• Strong sense of curiosity, patience, and collaboration — especially in working across different disciplines and cultures
• Familiarity with tools like LangChain, OpenAI API, Databricks, or similar LLM/AI development environments
• Experience designing evaluation frameworks or product analytics systems for AI-driven products
• Prior exposure to legal, financial, or corporate document datasets
• Experience mentoring or informally guiding junior technical teammates