AI vs Human Forecasting Wars

I have two groups that I meet with to discuss AI, one of which I host myself to see how we can use AI. The latest interesting discussion was how well AI does in forecasting. I’ve read often that AI does a better job of forecasting than humans, so I wanted to test that. This test eventually turned into an AI versus human thing when I brought along Yahoo Entertainment.

Test Methodology

A little background on the methodology before I expound on what I found.

I used CoPilot in this series of weekly tests only because it is more accessible and does not require login. Nothing special about CoPilot. I did think its web search capability might be its advantage.

The first series of tests started in January and ended sometime on January 17th. These series of tests were to see if CoPilot could pick out the potential films that could win the Best Picture category for the Oscars.

CoPilot managed to pick 7 out of 10 nominees for Best Picture 2025.

The next series were weekly queries of CoPilot, asking it to pick the top 3 contenders for Best Pictures out of the 10 provided to it and to give its confidence level of its assessment. This is basically tracking for 6 weeks its best guesses of top 3 potential winners.

At the same time, I had asked CoPilot to name those who has historically done the best at predicting the Oscar winners. CoPilot came back with Gold Derby, Variety, and Yahoo Entertainment.

Since Yahoo Entertainment provided a weekly tracking of how the films were being ranked out in the wild, I decided to track its trackings.

Digging deeper into Yahoo’s methodology, it looks like data analytics was at play, but that can be a bit gray here. CoPilot, ChatGPT and Google AI all seem to think no AI or machine learning was involved based upon its description of its methodology.

AI vs Human Results

The following slide presentation provides very briefly the results of this experiment. Hopefully the embedding of this slide works.

Here’s the main slide with THE takeaway from the experiment. I enlarged the view in the hopes you can see better.

It looks like Yahoo’s data analytics overlaid with human instincts/judgement latched on to a clear winner by early February. CoPilot zigged and zagged.

Of course, that could stem from the problem of my prompts.

Latest Study on AI vs Humans

Yes, there is a study from Wharton that I managed to find on Google AI after trying to find a quote about AI being better than humans. While it appeared there were no notable quotes, Google AI did provide me with a link about Wharton’s January 2025 paper on its research on forecasting.

A summary of its major findings is seen on page 11 of the slide:

The link to Wharton’s paper is here.

Conclusion: Humans still rule!

Well, to some extent. The best version, other than the superforecasters, is the averaging of the AI models with human instincts.