Which Teams Overachieved and Which Teams Underachieved in 2012?

Tonight, while working putting together the final validation of the 2013 FBS Prediction Model, my analysis of the results highlighted some very interesting, and kind of disconcerting, things about Nebraska's 2012 season.

The model simulates every FBS game for the entire season.  

As validation, I used the methodology to simulate the 2012 season and prepared a comparison between between the model results and the real game results.  

This first chart shows some of those results of the predicted versus actual wins. 

A positive number of the vertical axis means the model predicted more wins than a team achieved.  Likewise, the negative numbers mean the model predicted fewer wins than the team achieved.

The average delta between the predicted win total and the actual win total is 1.15 games per team. 

21% of teams were predicted correctly.  20% were under-predicted by one win, and 29% were over-predicted by one win.  The model predicted more than 70% of seasons accurately to within one win.

The model under-predicted 12% of season by two wins and 5% by three wins.  It over-predicted 7% over season by two wins and 6% of season by three wins.

Can you guess which team was the most over-achieving?  

You betcha. Dear Old NU.  

The model predicted 6.91 wins but the Huskers finished with 10 wins, for a delta of -3.09 wins.  

While some may be inclined to see this as a positive, and it's certainly better than winning three fewer games than the model anticipated, the fact remains that Nebraska's scoring offense and scoring defense (the basis of the model) should have resulted in a 7-win season...not a 10-win season.

Remember the amazing streak of 4th quarter heroics in the Wisconsin, Northwestern, and Michigan State, and Penn State games?  Remember how Denard Robinson left the game before halftime?   

Sometimes, it's better to be lucky than good.

Interestingly, Ohio State is the #2 most overachieving team, with a predicted win total of 9.12.  Had they played in the Big Ten Championship Game or a bowl game, they would almost certainly have been the most overachieving team by a large margin.

The Top-10 overachievers and Bottom-10 underachievers are:

I put a table with the full results of the model at the end of this post.

Finally, the model provides some insight into the consistency of a team's on-field performance.  The standard deviation of the predicted wins can be used as a proxy for a team's consistency.

So, can you guess which team had the largest standard deviation in the model results, and by proxy, was the least consistent and predictable?  

Yup, Dear Old NU.  Again.

Now, guess which team had the lowest standard deviation in model results.  

Hint:  they won the National Championship.

I'll let my readers draw their own conclusions about this one.

By the way, I'm now a writer over at Football Study Hall.  Stop by and check it out.  I'll be writing about more than just the Huskers over there.

GBR!
@HuskerMath

Rank	Team	Conf	Pred. Wins	Actual Wins	Delta (rnd)	Pred. Delta	Abs( Pred Delta)
1	Northern Illinois	MAC	13.1	12	1	1.1	1.1
2	Alabama	SEC	12.7	13	0	-0.3	0.3
3	Florida State	ACC	12.5	12	1	0.5	0.5
4	Utah State	WAC	11.2	11	0	0.2	0.2
5	Oregon	Pac-12	11.0	12	-1	-1.0	1.0
6	UCF	C-USA	11.0	10	1	1.0	1.0
7	Georgia	SEC	10.9	12	-1	-1.1	1.1
8	Boise State	MWC	10.8	11	0	-0.2	0.2
9	Arizona State	Pac-12	10.7	8	3	2.7	2.7
10	Cincinnati	Big East	10.6	10	1	0.6	0.6
11	Tulsa	C-USA	10.5	11	0	-0.5	0.5
12	BYU	Ind	10.5	8	2	2.5	2.5
13	Texas A&M	SEC	10.4	11	-1	-0.6	0.6
14	Kansas State	Big 12	9.9	11	-1	-1.1	1.1
15	North Carolina	ACC	9.8	8	2	1.8	1.8
16	Rutgers	Big East	9.8	9	1	0.8	0.8
17	Clemson	ACC	9.8	11	-1	-1.2	1.2
18	Oklahoma State	Big 12	9.8	8	2	1.8	1.8
19	Bowling Green	MAC	9.7	8	2	1.7	1.7
20	Ohio	MAC	9.7	9	1	0.7	0.7
21	Vanderbilt	SEC	9.7	9	1	0.7	0.7
22	San Jose State	WAC	9.6	11	-1	-1.4	1.4
23	Fresno State	MWC	9.6	9	1	0.6	0.6
24	Stanford	Pac-12	9.6	12	-2	-2.4	2.4
25	Kent State	MAC	9.5	11	-1	-1.5	1.5
26	Wisconsin	Big Ten	9.4	8	1	1.4	1.4
27	South Carolina	SEC	9.2	11	-2	-1.8	1.8
28	Notre Dame	Ind	9.2	12	-3	-2.8	2.8
29	Arkansas State	Sun Belt	9.2	10	-1	-0.8	0.8
30	Ohio State	Big Ten	9.1	12	-3	-2.9	2.9
31	San Diego State	MWC	9.1	9	0	0.1	0.1
32	Penn State	Big Ten	8.8	8	1	0.8	0.8
33	Florida	SEC	8.7	11	-2	-2.3	2.3
34	LSU	SEC	8.7	10	-1	-1.3	1.3
35	Oregon State	Pac-12	8.7	9	0	-0.3	0.3
36	Louisiana-Lafayette	Sun Belt	8.6	9	0	-0.4	0.4
37	Northwestern	Big Ten	8.6	10	-1	-1.4	1.4
38	Oklahoma	Big 12	8.5	10	-2	-1.5	1.5
39	Michigan	Big Ten	8.3	8	0	0.3	0.3
40	UCLA	Pac-12	8.2	9	-1	-0.8	0.8
41	Louisiana-Monroe	Sun Belt	8.2	8	0	0.2	0.2
42	Pittsburgh	Big East	8.1	6	2	2.1	2.1
43	Western Kentucky	Sun Belt	8.1	7	1	1.1	1.1
44	Louisville	Big East	8.1	11	-3	-2.9	2.9
45	Mississippi State	SEC	8.1	8	0	0.1	0.1
46	Rice	C-USA	7.9	7	1	0.9	0.9
47	TCU	Big 12	7.8	7	1	0.8	0.8
48	SMU	C-USA	7.8	7	1	0.8	0.8
49	Louisiana Tech	WAC	7.7	9	-1	-1.3	1.3
50	USC	Pac-12	7.7	7	1	0.7	0.7
51	Nevada	MWC	7.6	7	1	0.6	0.6
52	Michigan State	Big Ten	7.5	7	1	0.5	0.5
53	Syracuse	Big East	7.4	8	-1	-0.6	0.6
54	North Carolina State	ACC	7.4	7	0	0.4	0.4
55	Toledo	MAC	7.2	9	-2	-1.8	1.8
56	East Carolina	C-USA	7.2	8	-1	-0.8	0.8
57	Navy	Ind	7.2	8	-1	-0.8	0.8
58	Air Force	MWC	7.2	6	1	1.2	1.2
59	Georgia Tech	ACC	7.1	7	0	0.1	0.1
60	Texas	Big 12	7.1	9	-2	-1.9	1.9
61	Texas Tech	Big 12	7.0	8	-1	-1.0	1.0
62	Nebraska	Big Ten	6.9	10	-3	-3.1	3.1
63	Western Michigan	MAC	6.8	4	3	2.8	2.8
64	UTSA	WAC	6.6	8	-1	-1.4	1.4
65	Central Michigan	MAC	6.6	7	0	-0.4	0.4
66	New Mexico	MWC	6.5	4	3	2.5	2.5
67	Virginia Tech	ACC	6.4	7	-1	-0.6	0.6
68	Connecticut	Big East	6.4	5	1	1.4	1.4
69	Iowa State	Big 12	6.4	6	0	0.4	0.4
70	Washington	Pac-12	6.3	7	-1	-0.7	0.7
71	Middle Tennessee	Sun Belt	6.3	8	-2	-1.7	1.7
72	Ball State	MAC	6.3	9	-3	-2.7	2.7
73	Baylor	Big 12	6.2	8	-2	-1.8	1.8
74	Mississippi	SEC	6.2	7	-1	-0.8	0.8
75	Minnesota	Big Ten	6.2	6	0	0.2	0.2
76	Troy	Sun Belt	6.1	5	1	1.1	1.1
77	Utah	Pac-12	6.1	5	1	1.1	1.1
78	Memphis	C-USA	5.9	4	2	1.9	1.9
79	Iowa	Big Ten	5.6	4	2	1.6	1.6
80	Miami (Florida)	ACC	5.4	7	-2	-1.6	1.6
81	West Virginia	Big 12	5.3	7	-2	-1.7	1.7
82	Purdue	Big Ten	5.2	6	-1	-0.8	0.8
83	Arizona	Pac-12	5.2	8	-3	-2.8	2.8
84	Texas State	WAC	5.1	4	1	1.1	1.1
85	Houston	C-USA	5.1	5	0	0.1	0.1
86	Wyoming	MWC	4.9	4	1	0.9	0.9
87	South Alabama	Sun Belt	4.8	2	3	2.8	2.8
88	Marshall	C-USA	4.8	5	0	-0.2	0.2
89	UNLV	MWC	4.8	2	3	2.8	2.8
90	Virginia	ACC	4.8	4	1	0.8	0.8
91	Maryland	ACC	4.8	4	1	0.8	0.8
92	North Texas	Sun Belt	4.7	4	1	0.7	0.7
93	Colorado State	MWC	4.7	4	1	0.7	0.7
94	Buffalo	MAC	4.6	4	1	0.6	0.6
95	Indiana	Big Ten	4.5	4	0	0.5	0.5
96	Tennessee	SEC	4.4	5	-1	-0.6	0.6
97	UTEP	C-USA	4.4	3	1	1.4	1.4
98	Florida International	Sun Belt	4.3	3	1	1.3	1.3
99	UAB	C-USA	4.3	3	1	1.3	1.3
100	Duke	ACC	4.2	6	-2	-1.8	1.8
101	Akron	MAC	4.0	1	3	3.0	3.0
102	Army	Ind	4.0	2	2	2.0	2.0
103	South Florida	Big East	3.9	3	1	0.9	0.9
104	Temple	Big East	3.9	4	0	-0.1	0.1
105	Boston College	ACC	3.8	2	2	1.8	1.8
106	Florida Atlantic	Sun Belt	3.8	3	1	0.8	0.8
107	Missouri	SEC	3.5	5	-2	-1.5	1.5
108	Arkansas	SEC	3.5	4	-1	-0.5	0.5
109	Auburn	SEC	3.4	3	0	0.4	0.4
110	Wake Forest	ACC	3.3	5	-2	-1.7	1.7
111	Miami (Ohio)	MAC	3.3	4	-1	-0.8	0.8
112	Washington State	Pac-12	3.1	3	0	0.1	0.1
113	California	Pac-12	2.8	3	0	-0.2	0.2
114	Hawaii	MWC	2.7	3	0	-0.3	0.3
115	Kentucky	SEC	2.5	2	1	0.5	0.5
116	New Mexico State	WAC	2.4	1	1	1.4	1.4
117	Illinois	Big Ten	2.4	2	0	0.4	0.4
118	Eastern Michigan	MAC	2.3	2	0	0.3	0.3
119	Tulane	C-USA	1.8	2	0	-0.2	0.2
120	Kansas	Big 12	1.8	1	1	0.8	0.8
121	Southern Mississippi	C-USA	1.7	0	2	1.7	1.7
122	Idaho	WAC	1.3	1	0	0.3	0.3
123	Colorado	Pac-12	0.9	1	0	-0.1	0.1

Placing Fumbles in Context (Part 2)

Continuing my breakdown of fumbles, Part 2 looks at fumbles by distance to go, player position, and quarter.

Distance (to go)

Looking at the entire football field, the breakdown of fumbles by down and distance looks like this (fumbles on kickoffs and punts are excluded):

1st and 10 accounts for the overwhelming majority of fumbles, but it accounts for the lion's share of the down-distance pairings during a game, so there's nothing particularly surprising in that.  

For distances of 10 or greater, 12% of fumbles occur on 2nd down, 5% on 3rd down, and less than 1% on 4th down.  

For distances of fewer than 10 yards, 2nd down accounts for 21%, 3rd down accounts for 17%, and 4th down accounts for 3% of fumbles.

Because the fumble percentages correspond closely to the actual play distribution by distance to go in a football game I'm led to conclude that distance to go is not a significant contributing factor to the probability of a fumble occurring on a play.

Player Position

Across the FBS, QB's accounted for about 50% of fumbles in the opponent red zone. The percentage of QB fumbles decreased steadily as the team approached the end zone. Running backs' percentage of fumbles increased steadily as a team approached the end zone. WRs were most likely to fumble in the middle of the field. Interestingly, DBs accounted for a not-insignificant number of fumbles. I suppose this is following interceptions or fumble recoveries.

Powered by Tableau

When I look at this chart, absolutely nothing important jumps out at me. While there is some variation in frequency of fumbles between quarters, there's no reason to think that it is due to any reason other than chance.  

Conclusion

And this concludes my breakdown of fumbles.  If there's a useful takeaway from Parts 1 and 2, I think it is the improbable frequency of fumbles on punt returns.  

Follow @HuskerMath
Tweet

What can and can't sports analytics do?

Andrew Sharp at SBNation has a great article called Paralysis by Analysis in which he details his visit, as a confessed analytics skeptic, to the the MIT Sloan Sports Analytics Conference.  This conference, to guys like me, is like making the Hajj to Mecca for the world's muslims.  It has to be done, but everybody knows its damn expensive, so Allah (or in my case, Nate Silver) understands if it doesn't work out.

It got me thinking, along with a negative comment left by a reader this week, that some folks are misunderstanding what I'm trying to do, and what sports data and statistics analysis can do and can't do.

What can't sports analytics do?  It can't predict what's going to happen on the next play, series, inning, snap, or whatever.  It can't explain WHY something happened. And it can't take the place of a coach's experience.

What can sports analytics do?  It can provide insights into aspects of the game that are not readily apparent to someone watching, coaching, or browsing the box scores. It can serve as an early warning to coaches and managers about potential problem areas and trends before they manifest themselves in the box score (at which time it's probably too late).  And it can function as a way to evaluate players and coaches in a (mostly) objective manner.

The negative comment I mentioned above said this:  

After all of that it means really nothing...You still cannot prevent these kind of mistakes, and you surely will never be able to look at these graphs and charts, and decide before the next play "the fumbles a coming, better tell so and so to hang on to the ball".....Pretty much a big ole waste of time.....

The comment was directed at the first part of a two-part piece on fumbles that I wrote earlier this week.  I appreciate the commenter's feedback, but I think he's missing the point.  Or maybe I failed to help him understand the point.

That piece wasn't about saying "this play will result in a fumble".  It was about digging into the limited data available to identify relationships between separate events that might be exploitable.  What I found was that fumbles on punt returns occur far more often than they should if they happened at the same frequency as punts.  They don't, and that is an exploitable nugget of information.  A coach could take that to heart and realize that he needs to place more emphasis (read: time, practice, and coaching) into the act of catching and returning a punt.

Whether the analytics are the low budget work I'm doing or the amazing technology gathering and analysis that companies who went to the Sloan conference are engaging in; we are trying to do the same thing...uncover the hidden information in the game so coaches and players and make better informed decision.

GBR!

Paul

Follow @HuskerMath Tweet