
AI models agreed with each other more than with reality in a new test
CryptoSlate
CryptoSlate
AI models agreed more closely with one another than with laboratory measurements in a new test of AI predictions. Pooling their rankings improved results on average, but several systems could still share the same mistake.
The Science Prediction Universe study, led by UniPat AI and submitted to arXiv on Nov. 11, tested 13 agent configurations spanning 12 models on 100 public tasks and 16 private tasks. Each configuration combined a model, reasoning setting and agent software. The study asked them to rank specified experiments across chemistry, materials science, biology and medicine.
Science Prediction Universe researchers reported that, on the public tasks, average agreement between scored configurations reached a Spearman rank correlation of 0.52, compared with 0.33 against measured outcomes. Agreement between agents exceeded their correlation with measurements on 81 of the 100 tasks. These averages use available scored predictions, with coverage varying by configuration.
Those figures describe how closely orderings match, rather than percentages of answers correct. Agents might rank reaction conditions by yield or materials by conductivity without having to predict numerical values. A perfect ordering without measurement ties scores 1; random rankings average 0.
Consensus improved predictions
The researchers found a benefit to pooling predictions. Averaging each experiment's position across configurations produced a consensus ranking that scored 0.08 points above the average submission, improving results on 73 of the 100 public tasks. The reported 95% confidence interval for the gain was 0.05 to 0.11.
On one oncology task, eleven configurations agreed at 0.9 but averaged just 0.27 against measurements. The benchmark’s best individual system also beat the consensus of its peers on matched tasks.
More broadly, tasks with greater agreement tended to have better predictions. The paper's agreement analysis therefore presents consensus as an informative but fallible signal. The authors explain that averaging can reduce errors that differ between systems while retaining common ones.
The authors connect those shared errors to simple assumptions about how experimental outcomes should rise or fall. If several agents rely on the same mistaken relationship, collecting more of their opinions can preserve that error.
The configurations were scored on different task sets. Claude Opus 5.5 and Claude Fable 5.1 had scores on only 44 and 30 public tasks, respectively, largely because of safety refusals. Across the full benchmark, all 13 configurations shared just 28 scored tasks: 27 public and one private.
The public tasks also looked backward at published measurements and were selected after pilot evaluations. The private set contained only 16 tasks across four unpublished experimental series, limiting how broadly its findings can be applied.
There was useful predictive ability in that private set: twelve of the 13 configurations ranked the unpublished experiments significantly better than chance overall, according to UniPat's Nov. 12 results release. Of 208 official private runs, 159 were scored.
The study used no human prediction baseline. Its findings apply to ranking the experimental outcomes it tested, with lab measurements still needed to check the answers.
AI Summary
A new study by UniPat AI tested 13 AI agent configurations on scientific prediction tasks, finding they agreed more closely with each other than with actual laboratory measurements.
Sentiment analysis
AI insights
low
2d ago
general
Financial analysis
Relevance
medium
Market sensitivity
medium
Affected sectors
Affected assets
Follow this story on TV
Live charts and AI commentary on one screen with Trade on tv.