Is the "Close Game - Clutch Play" Story Backwards?
by Jim Glass
Do Teams that Lose Clutch Games The Win More in the Playoffs -- and that Win Clutch Games, Lose More?
"Great teams win close games" they say. If so, the Falcons are sitting pretty. But is it true? Does winning close games during the regular season foretell victory in the playoffs? Being a sports stats geek I thought I knew -- but before running the numbers I had no idea the answer is both so counter-intuitive and stark.
The argument: To win close games takes clutch play, and it is clutch play that shows the skill and character a team needs to win tough games on the way to a championship. Think of Joe Montana throwing "the pass". Everyone knows this, it is common sense. So winning close games in the regular season gives confidence in post-season success.
But wait, says the sports stats geek, it's not so obvious. If team B thumps the same opponents that team A does by bigger margins, the larger scores it runs up shows it's better than A. That's common sense too.
Not so, says Mr. Clutch Play. If B is better why doesn't it win more games? Wins are all that count. If B scores more without winning more then it isn't better -- it is wasting points, inefficient, lacks character. In the playoffs the opposition is tougher, the scores are closer, and the proven ability to win close games delivers victory ... The stats geek may make some response to this, but what's the point? At best the result is an infinite loop.
THE TEST
To break the loop I looked at the records of all NFL teams that made the playoffs after winning 11 or more games during the regular season over the last 15 years, 1995 to 2009. There are exactly 100 teams in this pool, as it happens. The object, simply enough, was to compare the playoffs records of the teams that won the most close games during the regular season to those that won the least.
In particular, the idea was to compare teams with the same regular season w-l records -- 12-4 teams to 12-4 teams, 13-3 to 13-3, etc. -- to see if the number of close ("clutch") wins and losses ("failures in the clutch") a team has portends anything for it in the playoffs, hoping that a clear pattern emerges. It does!
NFL games are decided by a median margin of 10 points, so I defined a "close game" as one decided by 10 points or less, and those decided by 11 or more as a "big win" or "big loss". (There's a reason for picking 10 points as the divider, apart from the nice 50-50 split it gives, that's discussed below.) The rest follows simply...
RESULTS
To begin with, gross number results...
Most close wins:
* Of the 100 teams, nine won nine-or-more close games during the regular season. Their subsequent record in the playoffs: 8-9.
* The 15 teams with the best records in close games during the regular season were a combined 103-11, 90%, in them. In the postseason they had a record of 15-14, 52%. (One Super Bowl was won among them -- but random chance would predict 2 SB winners among 15 teams.)
So it doesn't look like winning close games in the regular season gives any edge at all above 50-50 in the postseason -- but for believers in winning close games it gets worse!
Fewest close wins:
* Only seven teams of the 100 had losing records in close games during the regular season, going 18-30, 38%. In the playoffs these seven "very worst" close-game teams went 14-4, 78%, and won three Super Bowls.
Do you want your team to *lose* close games to win a championship!?
Super Bowl winners:
Let's look at things from the side of the 14 SB winning teams (the 10-6, 2007 Giants didn't make the cut). Their W-L records in...
* The playoffs, 46-0, 100% (obviously).
* The regular season, 175-49, 78%
* Close games during the regular season, 62-36, 63%
* Big wins/losses during the regular season: 113-13, 90%.
The 63% indicates some ability to win close games, but not a lot -- and even less than it seems. All the 100 teams averaged a 71% win rate in close game, so the SB winning teams were a little *worse* at winning close games than rest.
On the other hand, the 90% wins in "big" decisions looks like it might be a better indicator of playoff success than regular season W-L. Back to this in a bit.
Playoff results by win cohort:
By regular season W-L record, here's how winning close games matched up with winning playoff games. Teams in each "win cohort" (11, 12, etc.) are divided into a high and low group by close-game win percentage. Given are the number of teams in each group, their average win pct in close games, and their playoff record. (The three 16-0 and 15-1 teams are omitted for being too small a group):
14-2 (nine teams)
top 4 (94%): 8-3, 1 SB winner
low 5 (78%): 8-3 2 SB winners
13-3 (25 teams)
top 12 (86%): 7-12
low 13 (70%): 16-10 3 SB winners
12-4 (28 teams)
top 14 (75%) : 17-12 2 SB winners
low 14 (61%) : 21-10 4 SB winners
11-5 (35 teams)
top 20 (71%): 15-19 1 SB winner
low 15 (51%): 19-14 1 SB winner
The lower, "less clutch" groups win consistently.
* The "higher halves" had a season record in close games of 343-103, 77%, but a postseason record of only 47-46, 51%.
* The "lower halves" had a worse season record in close games of 229-142, 62%, yet a better postseason record of 64-37, 63%.
Go figure. Maybe you really *do* want your team to be poor at winning close games to win a championship.
Big wins and big losses.
Mentioned above is the idea that record in "big" wins and losses may be a better indicator of future playoff performance than regular W-L record. Now it seems even more logical. Record in big wins and losses is the flip side of record in close games -- if two teams have the same w-l record, and one won its games "closer", then the other won its games "bigger". Since close game w-l record seems to be a negative indicator of playoff success, you'd expect "big game" w-l record to be a positive one. But how much so?
* Four teams had 10 or more "big" wins in a season. Their playoff record was 11-1 with three SB wins (the fourth team was the 18-1 Patriots).
* Twelve teams had 9 or more "big" wins. Their record was 24-7 with five SB winners, plus three SB runners-up. So of these 12 teams 8 were in the Super Bowl.
I could continue, but you see the point: "big" wins are really indicative of playoff success.
From here one can quickly propose a simple W-L rating system that is indeed more effective at predicting future w-l than the standard w-l percentage. Count big wins and big losses, and add in close games as ties worth half a win. So a team with 6 big wins, 2 big loses and 4 close games is 8-4, .667. I'd call this a power rating but that name's already being used by everybody, so for the moment I'll call it a KickButt percentage -- the percentage of games a team has won big.
Here's how regular W-L pct compares to KickButt pct in playoff predictive power. Among the 100 playoff teams...
• The top 37 teams by regular W-L pct had records of 13-3 or better (total: 497-95, 84%) and in the playoffs they produced a record of 43-31, 58%.
• Just the top 24 teams by KickButt rating, 11.5-4.5 or better (total: 289-95, 75%), produced the same 43 playoff wins against only 15 losses, a 74% winning pecentage.
CONCLUSION
Maybe the most interesting to me of all the posts here at Advanced NFL Stats is the one showing half of NFL game outcomes are determined by luck. I'd always known there was a lot of chance in football, but *how much* was a point of endless argument and unsure opinion. To see it credibly quantified ... wow.
That was origin of using 10 points, the median w-l point differential in NFL games, in defining what's a close game. Half of all games are determined by luck, and it's reasonable to believe most of those games are the closest ones, so if you treat the 50% of games that are closest as ties you eliminate most of the luck. What remains is a W-L record much more solidly on the merits.
With all the talk here about "Atlanta" lately, I thought that using this idea to get a fix on how teams that win a lot of regular-season close games perform in the playoffs could be interesting. I expected to find that close games were determined mostly by luck in the critical moments, and a little by the quality of the team. Super Bowl champions winning 63% of close games fits well with that. I also expected that a Kickbutt pct would have better predictive value than normal W-L pct.
But I did *not* expect to find that the best teams in the playoffs do *worst* in close games, and the teams that do *worst* in close games do the best in the playoffs. Winning close games shows a weakness? That seems counter-intuitive.
I can think of three possible explanations:
1) Error. Whenever one reaches a conclusion that contradicts common sense and common wisdom the first possibility to consider is "mistake". I've done all this quick and dirty on the fly, I may have screwed up. Feel free to double-check, all.
2) Statistical bias. Half of all games being determined by luck is a *lot* of games. Yet almost all the teams with 11 or more wins have a good positive number of close game wins -- only seven had losing records in them. Possibly *many* of these teams are in the playoffs only because they've been lucky in so many close games. Then the teams that haven't been as lucky but have the same W-L record must be *better*.
Those seven teams that had really bad luck in their close games, losing them on net, had to be really superior to still get 11+ wins in the season -- so in the post-season they killed the teams there on luck and collected three SB titles.
3) "Hermball", or the conservative coaching strategy: "play safe to just stay close to have a chance to win at the end with superior execution". Yet a close end is going to be determined mostly by chance anyhow, while this sacrifices the chance for big wins ... but as much as I hate Hermball, I don't think this adds up to enough. My guess is #2.
One final thought: Throwing out close games or treating them as ties to estimate a team's strength is nothing new, but doing so systematically based on an objective measure of luck in game outcomes, as far as I know, *is* new. Since the idea is based on an insight presented here, and "KickButt" maybe isn't the best of names, we could call it the Burke Index or Burke Percentage instead (if our host wants his name associated with such a thing).
I'd print KickButt/Burke ratings for Atlanta and the rest of the league, but this has gone too long already.
For all who have troubled to read all the way to here, thank you for you interest (and tolerance).
PS: Factoids for any who may be interested:
* The team among these 100 the with the best KickButt number was St Louis, 1999: 14.5-1.5 (13-0 in big games, 0-3 in close ones). The 16-0 Pats were second (13.5-2.5)
* The bottom team by the same measure was Miami, 2008: 7.5-8.5 (officially 11-5).
* The team that gained the most from close wins was Indianapolis, 2009: 9-0 in close games (5-2 in others, for a KB 9.5-6.5 v 14-2 official).
Posts
10 comments:
Awesome job. I wonder though if 10 points is too many to be called a close game. If the cutoff is reduced to 7 points or less, would the results be robust? Do the samples get too small? Perhaps it would be even more condemning for the clutch narrative.
I think it's mostly point #2
If you anchor the 12-4 teams, and order by 'close game record', you're also sorting by 'blowout record' and then showing that teams that win big more often are the better teams.
It might be interesting to use the blowout record as your group and see whether teams with the same blowout record but different close game records have a significant difference (of course, then you could easily just end up proving that teams with more regular seasons wins wins more often in the playoffs, so that's not much use either).
Wow, awesome article.
What happens if you put all the teams together? I mean plot the close game win percentage against something like playoff winning percentage (or whatever you want to measure post season performance). What does that correlation look like?
Here is what i am thinking:
Lets assume that the true strength of a team is measured by point differential. Each team would have a mean and a standard deviation of point differential potential. Good teams would have a mean of maybe +10 and a SD of maybe ...12? who knows. Perfectly average teams would have a mean of 0 and a SD of ...12 again (lets assume everybody has a perfectly average strength of schedule). I would expect good teams to win more of their close games by this model, because the games that turned out to be close are really the ones where they were unlucky to be that close to begin with. They should have won by 21 but had a couple of random turnovers and only won by 7. If you picture the bell curve of the point differential, this makes sense. Good teams can still lose close games but they "should" be more likely to win them, simply because their mean is positive.
When this is plotted (close game winning % against playoff winning %) I would expect to still see a correlation. It wouldn't be a 1 for 1 relation; i think most of the games lost by good teams would be close, but i would have predicted to see a positive correlation.
I am really just wondering if the result might still be just as convincing if all teams were thrown in together and a trend emerges.
Awesome article.
Really good article, and I was thinking along the same lines as Ian about anchoring teams by their W-L record.
This is pretty much just a rehash of the FO "guts and stomps" theory.
It might be interesting to adjust for strength of opponent. If the Super Bowl teams had "only" a 63% record in close games, it may just mean they played tough schedules.
Thanks for the kind words, people.
I have a lot of tests and ideas to try with this. The first found that the Big Wins/Loss ratio identified playoff winners significantly better than did Pythagarian calculations, which is curious because of course they are variations on the same idea (points for/against is a better predictor of future WL than past WL).
I'm going to strength-of-schedule the numbers, and so on -- and hope to have it all done in the next four weeks so we can all make money betting on the playoffs! All suggestions are welcome, and anybody who wants to double-check me, like I said, I'm doing this on the fly and am not certifying anything as mistake-free yet.
This is pretty much just a rehash of the FO "guts and stomps" theory.
Sure, but it's not like they invented the idea, Bill James documented it in great detail for baseball 20-odd years ago and others have applied it to a lot of sports. PFR.com pointed out that Bill Walsh's 49ers had a 41% record in one-score games, and Lombardi's Packers had a 50% record.
What I'm trying to do is quanitify empirically the effect for the NFL in a systematic way, which as far as I know nobody has done yet.
I wonder though if 10 points is too many to be called a close game.
It was the logical number to start with in light of your (startling) finding that half of NFL games are determined by luck, in that it's the median point differential deciding games (for 2009). But others are to be tested.
Hey really excellent and interesting post!
I wonder if you could forget having a "close game" cutoff at all and try plotting average point differential in games won against playoff win %. And then sort the results by W-L record in the regular season.
What this could allow us to do is compare for example a 11-5 team that dominated in games they won versus a 13-3 or 14-2 team that had some close wins. It may or may not add anything to this discussion but if you have the time, I think it would be worth looking at.
Thanks for your contribution. It will be cool to see if it can predict playoff performance.
I wonder if you could forget having a "close game" cutoff at all and try plotting average point differential in games won against playoff win %...
I did something like this quick-and-dirty for my own information using each team's average point differential (more exactly, Pythagorean expectation) to predict playoff w-l.
Point differential worked signficantly better than regular season W-L at predicting playoff W-L (which I expected because Pythagorean is supposed to work better than past W-L in projections) but signficantly worse than big-win/tie/big-loss "KickButt" pct (which surprised me, because I imagined KB% as a back-door route back to Pythagorean).
* The top 37 teams by regular season W-L pct had records of 13-3 or better (total: 497-95, 84%), in the playoffs they produced a record of 43-31, 58%.
* The top 24 teams by KB pct., 11.5-4.5 or better (total: 289-95, 75%), produced the same 43 playoff wins against only 15 losses, a 74% winning pecentage.
* The top 30 teams by Pythagorean won the same 43 playoff games and lost 23, 65%, smack in the middle between the other two.
My tentative conclusion is that Pyathagorean includes noise from close games which counting them as ties eliminates.
For a clear example of the logical reason why teams that lose close games would do better in the playoffs (and vice versa) consider this example...
Key fact: Our host says half of all NFL games are determined by luck, that's a lot! Now say 10-6 is the record needed to make the playoffs, and two teams do it. In each's 16-game schedule 8 games are determined by luck and 8 by merit.
Team A is a bit lucky and two of its "luck" games switch to wins, so it goes 6-2 in them (instead of 4-4). In its "merit" games it then need go only 4-4, 50%, for its 10-6.
Team B is a bit unlucky and two of its "luck" games switch to losses, so it goes 2-6 in them. It has to go 8-0 in its "merit" games, 100%, to make its 10-6.
Head-to-head, Team B should score 75% against Team A, in spite of their equal 10-6 records, because on the merits it is a 100-0 team playing a 50-50 team. (In four games versus A it should go 2-0 on merit and 1-1 on luck.)
If close games are determined by luck, then teams that win enough games to make the playoffs with the "gift" of winning several games on luck must be weaker than teams that win enough games to get in the playoffs in spite of the handicap of having much less luck.
Jim,
As a Falcons fan let me just say I completely agree with your last paragraph. It stands to reason, and your analysis above thoroughly proves, that a 14-2 team with a bunch of blowout wins is better than a 14-2 team with a bunch of close wins. For that reason, I wouldn't argue that the Falcons are playing better than the Patriots right now.
I think a more fair comparison that Jason starts to allude to above, though, would involve comparing, say, 14-2 teams with good luck in close games to 12-4 teams with bad luck in close games. If a 14-2 team with good luck in close games lost a couple more of those, they wouldn't be a 14-2 team with bad luck in close games. They would be a 12-4 team with average luck.
It sounds like you would hypothesize that 14-2 teams that won two extra games by a field goal would fare equally well in the playoffs as 12-4 teams that lost those extra couple of close games, while advocates of clutch play would say the opposite, so that would be an interesting piece of analysis to perform.
I think your kick-butt index could be a good place to start. You could, for example, compare how historically teams with the same kick-butt index but differing win-loss records have done in the playoffs.
Also, I would be interested to see the kick-butt rankings for this year if you are willing to share. Thanks for the post.
It makes sense that there is a negative correlation between winning close games in the regular season and winning in the playoffs.
By defining a cutoff of 11 wins, you create a selection effect. Teams win games because they are lucky and good, so this selection effect means that your sample will be biased towards lucky and good teams. Moreover, because a better team needs less luck to reach 11 wins, and a less good team needs more luck to reach 11 wins, you should find a negative correlation between luck and skill in your sample.
Since skill is predictive and luck is not, this means that luck in your sample is negatively correlated with playoff success. This is not because good luck causes poor playoff success, but because of the negative correlation between luck and skill.
Post a Comment
Note: Only a member of this blog may post a comment.