AI Research Scientist

OxSci is building a credit rating agency for science, focusing on improving the peer review process through AI and expert evaluation. The AI Research Scientist will lead research on evaluating AI reviewers, design rigorous meta-evaluations, and develop standards for assessing research quality.

Responsibilities:

• Own the research agenda on AI-reviewer evaluation. Track the frontier (AI-scientist, automated-review, LLM-as-a-judge, and scholarly-NLP literature), position our system against it, and decide what we measure next and why
• Design meta-evaluations that expose weaknesses, not just measure agreement. Build fine-grained, criticism-level evaluations of AI review agents (correctness, factual grounding, significance, sufficiency of evidence, hallucination rate, and venue/journal matching) that reveal where and why they fail, going beyond verdict-matching
• Run expert-annotation studies at scale. Design the protocols, rubrics, inter-annotator agreement, and statistics needed to compare AI and human reviewers credibly, including head-to-head evaluations against other AI review systems, and defend the numbers to a skeptical scientific audience
• Build a living taxonomy of AI-reviewer failure modes such as subfield blind spots, long-context degradation, over-anchoring, and spurious criticism, and turn each into a regression benchmark that guards against backsliding as models and prompts change
• Calibrate the combined rating. Define quality-scoring rubrics for human review reports and calibrate how expert and AI judgment fuse into a single, defensible rating: the core of what universities and publishers buy from us
• Close the loop. Translate benchmark findings into concrete improvements to our review agents (retrieval, context engineering, orchestration, model choice) and prove the gains with the same rigor you used to find the gaps