Rating the pundits: 2009 NFL preseason predictions
How accurate are all those preseason predictions about how the coming NFL season will unfold?
In an effort to find out, I collected a number of preseason “team power rankings” two days before the 2009 NFL regular season started in September. These ranking lists are compiled by columnists and pundits from media outlets like Sports Illustrated, Fox Sports, The Sporting News, and ESPN. In addition, I collected a fan-voted ranking from Yahoo Sports and the preseason Vegas odds to win the Super Bowl. As a baseline of sorts, I’ve also included the ranking for how the teams finished in the 2008 season.
Each team ranking from each list was compared to the final 2009 regular season standings (taken from this tentative 2010 draft order) by calculating the offset between the estimated rank to the team’s actual finish. For instance, ESPN put the Steelers in the #1 slot but they actually finished 15th in the league…so ESPN’s offset for the Steelers is 14. For each list, the offsets for all 32 teams were added up and divided by 32 to get the average number of places that the list was off by. See ESPN’s list at right for example; you can see that each team ranking in the list was off by an average of about 6.3 places.
Here are the offset averages for each list (from best to worst):
| Media outlet | Offset ave. (# of places) |
| CBS Sports | 5.6 |
| The Sporting News | 5.6 |
| USA Today | 5.6 |
| Vegas odds | 5.8 |
| Yahoo Sports | 5.9 |
| Sports Illustrated | 5.9 |
| ESPN | 6.3 |
| Fox Sports | 6.4 |
| 2008 finish | 7.3 |
The good news is that all of the pundits beat the baseline ranking of last season’s final standings. But they didn’t beat it by that much…only 1.7 places in the best case. A few other observations:
- All the lists were pretty much the same. Last place Fox Sports and first place CBS Sports differ by less than one place in their rankings. The Steelers and Patriots were one and two on every list and the bottom five were pretty consistent as well. All the pundits said basically the same thing; no one had an edge or angle the others didn’t.
- Nearly everyone was very wrong about the Steelers, Giants, Titans, Jets, Bengals, and Saints…and to a lesser extent, the Redskins, Bears, Vikings, and Packers. CBS Sports made the fewest big mistakes; their offset for the Bengals was only 4 places. The biggest mistakes were Fox Sports’ choice and the Vegas ranking of the Bengals to finish 28th (offset: 19).
- Among the top teams, the Colts, Eagles, and Patriots more or less fulfilled the hopes of the pundits; only Fox Sports and Sports Illustrated missed the mark on one of these teams (the Colts by 9 places).
- The two “wisdom of the crowds” lists, Yahoo Sports and the Vegas list, ended up in the middle, better than some but not as good as some others. I suspect that there was not enough independent information out there for the crowd to make a good collective choice; those two lists looked pretty much like the pundits’ lists.
- The teams who turned out to be bad were easier to pick than the good teams. The bottom five picks on each list were typically off by 3-5 places while the top five were off by more like 8-12 places (esp the Steelers and the Giants). Not sure why this is. Perhaps badness is easier to see than goodness. Or it’s easier for a good-looking team to go bad than it is for bad-looking team to do better.
For the curious, here’s the full Google Docs spreadsheet of numbers for all of the lists.
Methodology and notes: 1) I made an assumption about all these power ranking lists: that what the pundits were really picking is the final regular season ranking. That isn’t precisely true but close enough for our purposes. 2) I have no idea what the statistical error is here. 3) The 2010 draft order list isn’t a perfect ranking of how the teams finished, but it is close enough. 4) Using the final regular season records as the determining factor of rank is problematic because of the playoffs. By the end of the season, some teams aren’t trying to win every game because they’ve either made the playoffs or haven’t. So some teams might be a little bit better or worse than their records indicate. 5) The Vegas odds list was a rankng of the odds of each team making the Super Bowl, not the odds for the teams’ final records. But close enough. 6) The Sports Illustrated list was from before the 2009 pre-season started; I couldn’t find an SI list from right before the regular season. Still, it looked a lot like the other lists and did middlingly well.
Stay Connected