NFL Coach Quality: A Bayesian Approach To Approximating the Value of Coaches - UPDATED

by David Durschlag

You are currently viewing version 1 of this article. To view version two, please click here.

Summary

Evaluating NFL coaches is a difficult task, popular among fans and vitally important to franchises. This is a brief attempt at the task, using purely quantitative data.

Data

The numbers of regular season games each team won each year are treated as data points. No information beyond number of regular season wins was used.

While the "metagame" of the NFL continues to evolve, the data used herein is from 1993 onward, when the last Collective Bargaining Agreement was signed. While victories now come in different environments, they are all under (roughly) the same rules. Data from before this period could be skewed based on the different rules for control of players, so it was excluded.

Also excluded was the performance of any team in a year in which it had multiple head coaches. This was done to ensure that credit for a season was easy to assign.

In total, 107 coaches and 565 team-years of data were used.

Assumptions

• Each coach has a hidden "value". Better coaches have higher value.
• The number of games a team wins can be modeled as a draw from a normal distribution with a mean of the value of their coach, and an unknown variance dubbed "Season Variance". This variance is constant across all seasons for all coaches.
• The value of coaches is normally distributed across the population, with unknown mean and variance.

Process

The above assumptions were encoded as a model for BUGS, which was run for 10,000 iterations, then another 10,000 iterations.

Results

Coach quality had converged after 10,000 iterations.

The posterior distribution for system constants were as follows:

Constant	Mean	Standard Deviation
Season Variance	0.137	0.009
Coach Value Population Mean	7.698	0.191
Coach Value Population Variance	0.626	0.207

The posterior distributions for coach values were as follows:

Coach	Mean Value	Standard Deviation Value
Bill Belichick	10.100	0.633
Tony Dungy	9.948	0.671
Mike Tomlin	9.490	0.920
Bill Cowher	9.350	0.636
Mike McCarthy	9.344	0.865
John Harbaugh	9.303	0.972
Sean Payton	9.245	0.852
Marty Schottenheimer	9.209	0.686
Mike Smith	9.197	0.970
Andy Reid	9.180	0.656
Mike Holmgren	9.107	0.606
Wade Phillips	9.051	0.792
Mike Shanahan	8.970	0.606
Barry Switzer	8.833	0.957
Mike Martz	8.763	0.863
Jimmy Johnson	8.751	0.892
Mike Sherman	8.746	0.856
Tom Coughlin	8.626	0.608
Jeff Fisher	8.578	0.604
Jack Pardee	8.549	1.221
Dennis Green	8.507	0.684
Brian Billick	8.505	0.738
Lovie Smith	8.466	0.777
George Seifert	8.412	0.812
Marv Levy	8.397	0.905
Rex Ryan	8.380	1.020
Don Shula	8.375	1.017
Bill Parcells	8.372	0.694
Jon Gruden	8.370	0.692
Steve Mariucci	8.216	0.778
Bobby Ross	8.148	0.772
Jim Caldwell	8.105	1.009
Wayne Fontes	8.094	0.945
Jim Fassel	8.077	0.805
Brad Childress	8.066	0.908
Dick Vermeil	8.058	0.778
John Fox	7.978	0.743
Pete Carroll	7.967	0.935
Al Groh	7.954	1.210
Ken Whisenhunt	7.876	0.897
Mike Tice	7.845	0.944
Jim Mora	7.833	0.956
Gunther Cunningham	7.801	1.104
Gary Kubiak	7.768	0.849
Jason Garrett	7.766	1.203
Jack Del Rio	7.759	0.746
Hue Jackson	7.745	1.201
Tony Sparano	7.734	0.943
Jim L. Mora	7.731	0.960
Dave Wannstedt	7.722	0.698
Dan Reeves	7.705	0.707
Norv Turner	7.704	0.634
Marvin Lewis	7.695	0.748
Nick Saban	7.631	1.094
Bill Callahan	7.629	1.099
Joe Gibbs	7.616	0.946
Mike White	7.607	1.102
Jim Haslett	7.584	0.855
Ray Rhodes	7.546	0.887
Mike Singletary	7.479	1.092
Mike Mularkey	7.467	1.103
Art Shell	7.423	1.016
Todd Haley	7.403	1.022
Jerry Glanville	7.373	1.210
Chan Gailey	7.358	0.968
Tom Cable	7.311	1.119
Rich Brooks	7.307	1.100
Josh McDaniels	7.155	1.102
Dick Jauron	7.151	0.752
Tom Flores	7.143	1.092
Buddy Ryan	7.142	1.096
Steve Spurrier	7.141	1.099
Lindy Infante	7.139	1.119
Jim Zorn	7.135	1.094
Eric Mangini	7.124	0.906
Vince Tobin	7.107	0.957
June Jones	7.104	1.022
Dennis Erickson	7.098	0.864
Herman Edwards	7.090	0.787
Butch Davis	7.002	0.947
Sam Wyche	6.997	1.030
Scott Linehan	6.983	1.104
Kevin Gilbride	6.977	1.232
Jim E. Mora	6.975	0.958
Richie Petitbon	6.975	1.231
Joe Bugel	6.967	1.113
Lane Kiffin	6.962	1.231
Romeo Crennel	6.870	0.957
Gregg Williams	6.867	1.033
Raheem Morris	6.864	1.035
Ted Marchibroda	6.793	0.861
Mike Nolan	6.722	1.025
Dave McGinnis	6.717	1.030
Chuck Knox	6.654	1.123
Bruce Coslet	6.617	0.972
Dom Capers	6.597	0.789
Mike Ditka	6.573	1.050
Dave Campo	6.569	1.032
Dick LeBeau	6.496	1.124
Mike Riley	6.443	1.052
Cam Cameron	6.366	1.263
Dave Shula	6.311	1.041
Rich Kotite	6.263	0.995
Chris Palmer	6.011	1.162
Marty Mornhinweg	6.009	1.164
Steve Spagnuolo	5.877	1.072
Rod Marinelli	5.864	1.071

Conclusions and Analysis

Despite having a small sample size and extremely limited data, the estimated value of coaches agrees closely with conventional wisdom. Value estimates are relatively narrow, ranging from a 66% chance that Jeff Fisher's value is between 7.974 and 9.182 to a 66% chance Cam Cameron's value is between 5.103 and 7.629. Applying Microsoft's ? - 3 * ? method of combining the parameters of a normal distribution for TrueSkill, then renormalizing (so that the unit of value is approximately the win), the following list is obtained:

Coach	Normalized Combined Rating
Bill Belichick	10.060
Tony Dungy	9.870
Bill Cowher	9.520
Mike Holmgren	9.410
Andy Reid	9.360
Mike Shanahan	9.310
Marty Schottenheimer	9.310
Tom Coughlin	9.070
Jeff Fisher	9.040
Mike McCarthy	9.030
Mike Tomlin	9.010
Sean Payton	8.980
Wade Phillips	8.970
Dennis Green	8.820
John Harbaugh	8.770
Jon Gruden	8.700
Bill Parcells	8.700
Brian Billick	8.700
Mike Smith	8.700
Mike Sherman	8.620
Mike Martz	8.620
Lovie Smith	8.590
Jimmy Johnson	8.550
George Seifert	8.480
Barry Switzer	8.470
Steve Mariucci	8.410
Bobby Ross	8.380
Norv Turner	8.350
John Fox	8.320
Dick Vermeil	8.300
Marv Levy	8.270
Jim Fassel	8.260
Dave Wannstedt	8.230
Dan Reeves	8.200
Jack Del Rio	8.150
Marvin Lewis	8.110
Brad Childress	8.030
Don Shula	8.010
Rex Ryan	8.010
Wayne Fontes	7.970
Gary Kubiak	7.940
Ken Whisenhunt	7.920
Pete Carroll	7.900
Jim Caldwell	7.840
Jim Haslett	7.800
Mike Tice	7.790
Jim Mora	7.760
Tony Sparano	7.720
Dick Jauron	7.710
Ray Rhodes	7.700
Jack Pardee	7.700
Jim L. Mora	7.680
Joe Gibbs	7.630
Herman Edwards	7.590
Dennis Erickson	7.430
Gunther Cunningham	7.420
Chan Gailey	7.390
Eric Mangini	7.360
Art Shell	7.340
Nick Saban	7.320
Todd Haley	7.310
Bill Callahan	7.310
Al Groh	7.300
Mike White	7.290
Vince Tobin	7.240
Dom Capers	7.240
Ted Marchibroda	7.220
Mike Singletary	7.220
Butch Davis	7.190
Mike Mularkey	7.190
Jason Garrett	7.180
Hue Jackson	7.170
Jim E. Mora	7.150
June Jones	7.100
Rich Brooks	7.080
Romeo Crennel	7.070
Tom Cable	7.040
Sam Wyche	7.010
Tom Flores	6.980
Buddy Ryan	6.970
Jim Zorn	6.970
Josh McDaniels	6.970
Steve Spurrier	6.960
Lindy Infante	6.920
Gregg Williams	6.910
Raheem Morris	6.900
Jerry Glanville	6.890
Bruce Coslet	6.860
Scott Linehan	6.840
Mike Nolan	6.820
Joe Bugel	6.810
Dave McGinnis	6.810
Dave Campo	6.700
Mike Ditka	6.660
Mike Riley	6.570
Chuck Knox	6.560
Richie Petitbon	6.560
Kevin Gilbride	6.560
Rich Kotite	6.560
Lane Kiffin	6.550
Dave Shula	6.500
Dick LeBeau	6.450
Steve Spagnuolo	6.120
Rod Marinelli	6.110
Cam Cameron	6.060
Chris Palmer	6.020
Marty Mornhinweg	6.020

The top two coaches both spent a lot of their career with transcendent quarterbacks. With the firing of Steve Spagnuolo and Raheem Morris, Romeo Crennel is the lowest ranking active head coach. If Bill Cowher says he'd like to get back into coaching, and inquires as to whether your organization is hiring, you say "yes." Using a similar model based on points for/against might not only produce slightly more accurate information, but could also reveal whether head coaches can be "defense-oriented" or "offence-oriented" and, if so, which ones are which.

Summary

Evaluating NFL coaches is a difficult task, popular among fans and vitally important to franchises. This is a brief attempt at the task, using purely quantitative data. The goal here is dual -- both to increase our understanding of how (and how much) NFL teams are reflective of their coaches, as well as to introduce the dichotomy of frequentist and Bayesian analysis to the NFL statistics community, from which it is largely absent.

Data

The numbers of regular season games each team won each year are treated as data points. No information beyond number of regular season wins was used.

While the "metagame" of the NFL continues to evolve, the data used herein is from 1993 onward, when the last Collective Bargaining Agreement was signed. While victories now come in different environments, they are all under (roughly) the same rules. Data from before this period could be skewed based on the different rules for control of players, so it was excluded.

Also excluded was the performance of any team in a year in which it had multiple head coaches. This was done to ensure that credit for a season was easy to assign.

In total, 107 coaches and 565 team-years of data were used.

Assumptions

• Each coach has a hidden "value". Better coaches have higher value.
• The number of games a team wins can be modeled as a draw from a normal distribution with a mean of the value of their coach, and an unknown standard deviation which is constant across all seasons for all coaches. This will be refered to as "Season Standard Deviation."
• The value of coaches is normally distributed across the population, with unknown mean and standard deviation. These will be refered to as "Population Mean" and "Population Standard Deviation".

Process

The above assumptions were encoded as a model for BUGS, which was run for 10,000 iterations, then another 10,000 iterations.

Results

Coach quality had converged after 10,000 iterations.

The posterior distribution for system constants were as follows. Posterior information for individual coaches is included in the large table in the next section.

Constant	Mean	Standard Deviation
Season Standard Deviation	2.70	0.09
Population Mean	7.69	0.19
Population Standard Deviation	1.33	0.18

Conclusions and Analysis

Below is a table with a variety of data about each coach.

• Value Posterior Mean: The mean of the posterior distribution for the coach's value.
• Value Posterior StdDev: The standard deviation of the posterior distribution for the coach's value.
• Triple-Conservative Rating: μ - 3 * σ -- a conservative single-number rating. Microsoft uses this to collapse TrueSkill distributions to single ratings.
• Normalized Triple-Conservative Rating: The triple-conservative rating re-normalized to the same scale as posterior mean coach value.
• Raw Average Wins: The coach's raw number of average wins over the data set. Provided for comparison to the posterior means -- this is one way to gauge the benefit of the Bayesian model over a standard frequentist approach. Because the Bayesian model incorporates the idea of uncertainty, a coach with one season of 14 wins is not considered a 14 win coach. Alternately sorting between this column and that of posterior mean, then judging which list looks better, is a reasonable shortcut for juding the modeling approach taken here.
• 80% Confidence Range: The range of win values the model expects with 80% confidence. In other words, this coach would be expected to win fewer games than the minimum 10% of the time, and more than the maximum 10% of the time. This can be used to gauge the amount of information added by the model -- the narrower these ranges are, the more information was available for judging that coach. The fact that these tend to be very wide indicates that the model cannot make strong predictions -- a result of both the minimal amount of data available and the unpredictability of the NFL. Taking all coaches as a data set together yields an 80% confidence range of 4.27-11.91. Comparing this to the ranges for invidual coaches is a good way to see how much information the model was able to add. In general, you'll find that the range has narrowed only very slightly, but has shifted a win or two. This can be translated roughly as "the uncertainty created by minimal data and the general difficulty in predicting the NFL means it is hard to predict how many games an individual coach's team will win, but we have a pretty good idea who the 'better' coaches are."

Coach	Value Posterior Mean	Value Posterior StdDev	Triple-Conservative Rating	Normalized Triple-Conservative Rating	Raw Average Wins	80% Confidence Range
Al Groh	7.95	1.21	4.32	7.30	9	4.03-11.87
Andy Reid	9.18	0.65	7.21	9.36	9.69	5.81-12.54
Art Shell	7.42	1.01	4.37	7.34	7	3.70-11.14
Barry Switzer	8.83	0.95	5.96	8.47	10	5.17-12.49
Bill Belichick	10.1	0.63	8.20	10.06	10.8	6.76-13.43
Bill Callahan	7.62	1.09	4.33	7.31	7.5	3.82-11.43
Bill Cowher	9.35	0.63	7.44	9.52	9.85	6.00-12.69
Bill Parcells	8.37	0.69	6.29	8.70	8.63	4.97-11.77
Bobby Ross	8.14	0.77	5.83	8.38	8.37	4.67-11.62
Brad Childress	8.06	0.90	5.34	8.03	8.4	4.45-11.67
Brian Billick	8.50	0.73	6.29	8.70	8.88	5.06-11.94
Bruce Coslet	6.61	0.97	3.70	6.86	5.5	2.93-10.29
Buddy Ryan	7.14	1.09	3.85	6.97	6	3.34-10.94
Butch Davis	7.00	0.94	4.16	7.19	6.25	3.34-10.65
Cam Cameron	6.36	1.26	2.57	6.06	1	2.39-10.33
Chan Gailey	7.35	0.96	4.45	7.39	7	3.68-11.03
Chris Palmer	6.01	1.16	2.52	6.02	2.5	2.14-9.87
Chuck Knox	6.65	1.12	3.28	6.56	4.5	2.82-10.48
Dan Reeves	7.70	0.70	5.58	8.20	7.7	4.29-11.11
Dave Campo	6.56	1.03	3.47	6.70	5	2.83-10.30
Dave McGinnis	6.71	1.03	3.62	6.81	5.33	2.98-10.45
Dave Shula	6.31	1.04	3.18	6.50	4.33	2.56-10.05
Dave Wannstedt	7.72	0.69	5.62	8.23	7.72	4.31-11.12
Dennis Erickson	7.09	0.86	4.50	7.43	6.66	3.52-10.66
Dennis Green	8.50	0.68	6.45	8.82	8.81	5.11-11.89
Dick Jauron	7.15	0.75	4.89	7.71	6.88	3.69-10.60
Dick LeBeau	6.49	1.12	3.12	6.45	4	2.66-10.32
Dick Vermeil	8.05	0.77	5.72	8.30	8.25	4.57-11.54
Dom Capers	6.59	0.78	4.23	7.24	6	3.10-10.09
Don Shula	8.37	1.01	5.32	8.01	9.33	4.65-12.09
Eric Mangini	7.12	0.90	4.40	7.36	6.6	3.51-10.73
Gary Kubiak	7.76	0.84	5.22	7.94	7.83	4.21-11.32
George Seifert	8.41	0.81	5.97	8.48	8.85	4.89-11.93
Gregg Williams	6.86	1.03	3.76	6.91	5.66	3.12-10.60
Gunther Cunningham	7.80	1.10	4.48	7.42	8	3.99-11.61
Herman Edwards	7.09	0.78	4.72	7.59	6.75	3.59-10.58
Hue Jackson	7.74	1.20	4.14	7.17	8	3.83-11.65
Jack Del Rio	7.75	0.74	5.52	8.15	7.77	4.30-11.21
Jack Pardee	8.54	1.22	4.88	7.70	12	4.62-12.47
Jason Garrett	7.76	1.20	4.15	7.18	8	3.85-11.67
Jeff Fisher	8.57	0.60	6.76	9.04	8.81	5.26-11.88
Jerry Glanville	7.37	1.21	3.74	6.89	6	3.45-11.28
Jim Caldwell	8.10	1.00	5.07	7.84	8.66	4.39-11.82
Jim E. Mora	6.97	0.95	4.10	7.15	6.25	3.31-10.63
Jim Fassel	8.07	0.80	5.66	8.26	8.28	4.56-11.58
Jim Haslett	7.58	0.85	5.01	7.80	7.5	4.02-11.14
Jim L. Mora	7.73	0.96	4.85	7.68	7.75	4.06-11.39
Jim Mora	7.83	0.95	4.96	7.76	8	4.17-11.49
Jim Zorn	7.13	1.09	3.85	6.97	6	3.33-10.93
Jimmy Johnson	8.75	0.89	6.07	8.55	9.6	5.15-12.34
Joe Bugel	6.96	1.11	3.62	6.81	5.5	3.14-10.78
Joe Gibbs	7.61	0.94	4.77	7.63	7.5	3.96-11.26
John Fox	7.97	0.74	5.74	8.32	8.11	4.52-11.42
John Harbaugh	9.30	0.97	6.38	8.77	11	5.62-12.98
Jon Gruden	8.37	0.69	6.29	8.70	8.63	4.97-11.76
Josh McDaniels	7.15	1.10	3.84	6.97	6	3.34-10.96
June Jones	7.10	1.02	4.03	7.10	6.33	3.37-10.83
Ken Whisenhunt	7.87	0.89	5.18	7.92	8	4.27-11.47
Kevin Gilbride	6.97	1.23	3.28	6.56	4	3.03-10.91
Lane Kiffin	6.96	1.23	3.26	6.55	4	3.02-10.89
Lindy Infante	7.13	1.11	3.78	6.92	6	3.31-10.96
Lovie Smith	8.46	0.77	6.13	8.59	8.87	4.98-11.94
Marty Mornhinweg	6.00	1.16	2.51	6.02	2.5	2.13-9.87
Marty Schottenheimer	9.20	0.68	7.15	9.31	9.75	5.81-12.60
Marv Levy	8.39	0.90	5.68	8.27	9	4.78-12.00
Marvin Lewis	7.69	0.74	5.45	8.11	7.66	4.24-11.14
Mike Ditka	6.57	1.05	3.42	6.66	5	2.81-10.32
Mike Holmgren	9.10	0.60	7.29	9.41	9.5	5.79-12.41
Mike Martz	8.76	0.86	6.17	8.62	9.5	5.19-12.33
Mike McCarthy	9.34	0.86	6.74	9.03	10.5	5.77-12.91
Mike Mularkey	7.46	1.10	4.15	7.19	7	3.65-11.27
Mike Nolan	6.72	1.02	3.64	6.82	5.33	2.99-10.45
Mike Riley	6.44	1.05	3.28	6.57	4.66	2.68-10.20
Mike Shanahan	8.97	0.60	7.15	9.31	9.31	5.65-12.28
Mike Sherman	8.74	0.85	6.17	8.62	9.5	5.18-12.30
Mike Singletary	7.47	1.09	4.20	7.22	7	3.68-11.27
Mike Smith	9.19	0.96	6.28	8.70	10.75	5.52-12.87
Mike Tice	7.84	0.94	5.01	7.79	8	4.19-11.49
Mike Tomlin	9.49	0.92	6.72	9.01	11	5.86-13.11
Mike White	7.60	1.10	4.30	7.29	7.5	3.79-11.41
Nick Saban	7.63	1.09	4.34	7.32	7.5	3.83-11.43
Norv Turner	7.70	0.63	5.80	8.35	7.71	4.36-11.04
Pete Carroll	7.96	0.93	5.16	7.90	8.25	4.32-11.60
Raheem Morris	6.86	1.03	3.75	6.90	5.66	3.12-10.60
Ray Rhodes	7.54	0.88	4.88	7.70	7.4	3.95-11.13
Rex Ryan	8.38	1.02	5.32	8.01	9.33	4.65-12.10
Rich Brooks	7.30	1.1	4.00	7.08	6.5	3.50-11.11
Rich Kotite	6.26	0.99	3.27	6.56	4.75	2.56-9.96
Richie Petitbon	6.97	1.23	3.28	6.56	4	3.03-10.91
Rod Marinelli	5.86	1.07	2.65	6.11	3.33	2.08-9.64
Romeo Crennel	6.87	0.95	3.99	7.07	6	3.20-10.53
Sam Wyche	6.99	1.03	3.90	7.01	6	3.26-10.73
Scott Linehan	6.98	1.10	3.67	6.84	5.5	3.17-10.79
Sean Payton	9.24	0.85	6.68	8.98	10.33	5.68-12.80
Steve Mariucci	8.21	0.77	5.88	8.41	8.5	4.73-11.69
Steve Spagnuolo	5.87	1.07	2.66	6.12	3.33	2.09-9.65
Steve Spurrier	7.14	1.09	3.84	6.96	6	3.33-10.94
Ted Marchibroda	6.79	0.86	4.21	7.22	6.16	3.22-10.35
Todd Haley	7.40	1.02	4.33	7.31	7	3.67-11.13
Tom Cable	7.31	1.11	3.95	7.04	6.5	3.48-11.13
Tom Coughlin	8.62	0.60	6.80	9.07	8.87	5.31-11.93
Tom Flores	7.14	1.09	3.86	6.98	6	3.34-10.94
Tony Dungy	9.94	0.67	7.93	9.87	10.69	6.57-13.32
Tony Sparano	7.73	0.94	4.90	7.72	7.75	4.08-11.38
Vince Tobin	7.10	0.95	4.23	7.24	6.5	3.44-10.77
Wade Phillips	9.05	0.79	6.67	8.97	9.75	5.55-12.54
Wayne Fontes	8.09	0.94	5.25	7.97	8.5	4.44-11.74

One thing to note about this information is that some of it points to possible violations of the model's assumptions. The top two coaches both spent a lot of their career with transcendent quarterbacks, for example. That's ok -- the model will simply be off on coaches for whom the assumptions don't hold as well. This is not about finding an exact cause-and-effect relationship, it's about finding general corrolative information, which means that some coaches will be easier to predict than others.