skip to content
Prolific AI ResearchPAIR

Prolific AI Research (PAIR)

A flow field of living data — cells and streams of information drifting across the canvas.

The AI research group at Prolific — papers, notes, and field logs.

Featured

noteHuman evaluation

People don't favour sound AI health advice over flawed advice, and AI judges disagree about which is which

When one of two AI replies to a health question has a flagged clinical problem, members of the public pick it at least as often as the sound one; clinicians pick it about one time in five. Two AI models given identical instructions flag serious errors at very different rates. What 100 clinicians showed us, and what it means for measuring the safety of AI health advice.

HUMAINE is our study of how people experience AI. A member of the public talks to two anonymous models about whatever matters to them, then compares them and picks a winner. The sample is balanced across demographic groups in the UK and the US. Health is the largest single topic, 14.7% of all…

Read · 8 min→
LeaderboardsAll evaluations →
A field of seed-stems gathering into a tall cluster — many human judgements converging into a ranking.

Human evaluation

HUMAINE Leaderboard

A demographically-aware, multi-dimensional human-preference leaderboard: 27 models judged by 20,000+ stratified participants and ranked with a hierarchical Bradley–Terry–Davidson model. Compare models head-to-head, by metric, and across 22 demographic groups.

Open leaderboard ↗
A radial bloom over concentric contour rings, crossed by a single plumb-line — behaviour measured against a held boundary.

Alignment

Alignment Leaderboard

Behavioural alignment evaluated under realistic pressure — 904 multi-turn scenarios across Honesty, Safety, Non-Manipulation, Robustness, Corrigibility, and Scheming. Ranks how models actually behave when instructions conflict, not what they claim they would do.

Open leaderboard ↗
A heart-shaped leaf printed in two inks that do not quite line up, inside hand-drawn contour rings, with seedlings along the ground: what people trust and what is safe, examined closely.

Health evaluation

HUMAINE Health

How the public uses and judges AI for their health, and how the same models hold up when practising clinicians check the advice: where models' health advice goes wrong, what people ask and reward, and why AI judges need checking against clinicians.

Open HUMAINE Health ↗
WritingAll writing →

noteHuman evaluation

The State of HUMAINE

Preference prediction hits a wall at 66%, demographics explain about 1% of how people judge, and the crowd rewards substance over flattery 2.5 to 1. Yet the pooled signal still teaches sycophancy. A state-of-the-project report from 100,000+ human comparisons, and the studies we're running next.

noteAgentic research

When does autoresearch need a human?

We ran Karpathy's autoresearch loop on a DPO task, then handed the same model its results in a single Claude Code session. 300 Prolific participants judged the outputs: the autonomous loop plateaued below chance, while five minutes of human steering produced the only decisive wins.

noteHuman evaluation

HUMAINE: A Rigorous Framework for Understanding AI Through Human Experience

A demographically-aware human-preference framework: 20,000+ stratified participants, 27 models, and a hierarchical Bradley–Terry–Davidson model that turns 21,352 judgements into an interactive, multi-dimensional leaderboard.

PublicationsAll publications →
DatePaper