Thursday, January 31, 2013

Does the SEC's oversigning make a difference? Part 1

The SEC allows teams to oversign.  That is, teams can sign more recruits than they can allow on to the team per NCAA regulations.  The question I'd like to know is whether this practice has a material effect a team's season success.

This chart shows the number of recruiting classes signed, by conference and by number of recruits signed.


The numbers above the Conference Name is the average number of recruits signed for every team in the conference from 2005-2012.

The numbers in the chart matrix indicate the number of recruiting classes in a conference with a given number of signees.  For example, The SEC has a 7 in the cell for 'SEC' and '32'.  This means that in the SEC from 2005-2007, there were seven recruiting classes that signed 32 recruits. 

What does this mean, then?

The average number of recruits for the SEC was 24.8 per team, per season.  The next highest average was the Big12 at 23.4 recruits per team, per season.

A simple t-test on these two datasets indicates that there is a statistically significant difference between the average number of SEC recruits per class and the average number of Big12 recruits per class.  It follows that the differences for all other conferences and the SEC are also statistically significant.

So, the conclusion to be drawn is that the policy of oversigning makes a difference in the number of recruits per season, per team. 

But does it make a difference on the field?  That question will be much more complicated to answer, but it's the topic of my next HuskerMath blog post.

Recruiting Part Trois

This post is inspired by a poster to the Huskers List-Serve.



While there may be a low correlation between recruiting rankings and final WL%, as Skylar points out, there is a very obvious relationship in the other direction...teams which finish highly ranked have very high recruiting rankings.






The first chart is built from recruiting ranking data from Scout.com from 2005-2012.  It shows the average number of top-100 recruits, 5-stars, 4-stars, 3-stars, the average four-year moving average, and average total stars for each team which finished with an final AP rank of 1-25.

Color coding is (best to worst) dark green to dark red.  The bottom line, Top-25 average, is the middle yellow.

It doesn't take a rocket surgeon to realize that teams which finish very high in the AP, finish year over year, with very high recruiting rankings.

Teams finishing highly in AP rankings have the lion's share of Top-100 recruits, # of 5- and 4-star recruits, and the best 4-year recruiting rank average.

While there is no clear predictive use for recruiting rankings...simply being ranked highly in the Scout.com rankings is not highly correlated to a high AP finish...teams which finish highly ranked in the AP share one commonality:  They are loaded to the gills with talent.

The data shows that teams which rely primarily on 3-star athletes as the centerpieces of their recruiting are likely to finish in the middle to bottom of the final AP poll.   

If I expand the pool a bit, and look at all teams in the NCAA, the differences are even more starkly defined:



Aggregating teams by final AP ranking demonstrates that teams finishing highly ranked in the AP final poll have a clear advantage in recruiting.

So, what does it all mean?  I believe it means what most already know...teams which finish at the top of the polls tend to be loaded with talent...or at least with players that Scout.com considers the most prized recruits each year.  While it's important not to say that if a team recruits well it will finish at or near the top of the polls, it's less problematic to say that if a team finishes at or near the top of the polls it has recruited well.

Recruiting does matter.  One 5-star player will not make or break a season.  However, recruiting two or three 5-star every year, accompanied by six or seven 4-star recruits, is what top-5 teams do.  A high rank finish isn't guaranteed, but it's more likely than with a stable of 3-, 2-, and 1-stars and unranked recruits.


  If there is any further evidence needed, look at the averages for participants in the BCS National Championship Games for 2005-2012 (with the NCAA Average and AP Top 25 Averages below:

Here's the breakdown by team for all participants in the BCS National Championship Games:




Tuesday, January 29, 2013

Average 4-year Recruiting Rankings and WL%

To follow up on my last post about recruiting, I dug into Scout.com's historical recruiting rankings and pulled some interesting stuff out of the data.

First of all, when I do the same correlation computations that I wrote about in my previous post on all NCAA teams I find that no average or single year has a correlation coefficient of greater than .36.  This means that recruiting rankings alone are poorly correlated to season win percentage.  So, I'm abandoning my hopes of an expanded season model.

One thing you can do, without interpreting or extrapolating the data, is compute the probability, given a 4-year Scout recruiting rank average, of a team finishing with a particular season W/L percentage.

For example, for the 2005-2012 seasons, there were 56 seasons in which the Scout.com 4-year average was between 1 and 10.  11 of 56 (19.6%) resulted in a season with a WL% of > 90%.  This is reflected in the upper left corner of the probability matrix.


Figure 1

To show the most relative probability of a given season WL%, I color coded the rows with the most likely season WL% as darkest green and the least likely WL% as the darkest red.  One thing that jumps out at me is that a team is most likely to have a fairly average (between .41 and .70) WL% regardless of recruiting.  This is shown by the dominance of green in the center of the chart.  The chart does show that a very high recruiting average increases the likelihood of a very successful season and the likelihood that a very low recruiting average increases the likelihood of a low WL%


So, where did NU's recruiting averages end up?
 
NU's 4-year averages are shown in Figure 2:


Figure 2















Those averages, plotted on the chart in Figure 1, look like this:

Figure 3

Other than 2007, AKA "The Season That Shall Not Be Recalled", NU's recruiting and WL% has formed a fairly tight shot group.  The conclusion that I draw from this is that NU has actually slightly outperformed what the probabilities would predict; based on their average recruiting ranking.

At the risk of assuming the role of Major Obvious, if NU is going to make the jump into elite status (which I would consider a WL% of >80%), NU's recruiting has to be improved and that improvement needs to be sustained over a long period of time.  If NU's Scout.com 4-year average can move into the 1-10 group, there is 40% probability that NU's WL% will be >80%.  If it continues to average about 26 the probability of an elite season is <10%, and the probability of competing for a National Championship is close to 0. 

Taking each team's actual end of season WL% and dividing it by the expected WL% for that recruiting group, I can identify teams that over/underperformed that year...at as far a their recruiting ranking would predict.

For the Big 10, sorted by over to under performing over the entire period, it looks like this:




It looks to me like NU has, as I mentioned above, slightly outperformed what would be expected of their recruiting classes.  MSU's and NWU's recent successes show up clearly as well.  Michigan's average is pulled down by it's lousy 2008 and 2009 seasons. 

Friday, January 25, 2013

Is NU recruiting on solid ground?

Brandon Vogel at Hail Varsity has a good article about the history of NU recruiting rankings over the last 25 years.

He maintains that NU is in a good position with our current recruiting.

I think that he's being a bit optimistic.

Using the recruiting ranks he posted and the W/L% for each year, I built a chart to determine the best correlation between recruiting rankings and W/L%. I used the current year ranking (what Brandon posted) and 2,3,4,5, and 6 year moving averages.


What I found was that best correlation between ranking and a year's W/L% is a 6-year moving average.




Current Year6 year average5 year average4 year average3 year average2 year average
Season W/L%-0.390-0.579-0.513-0.500-0.514-0.483


Using current year recruiting averages had the worst correlation, which is not surprising...what is surprising is that recruiting classes from 5 and 6 years prior to the current year could still have an effect on the season outcomes. Whether this is 'culture of winning' or something else merits further examination I think.

Now, knowing that a 6-year moving average is the strongest correlation to season W/L%, I can compute a regression equation based on the 6-year moving averages between 1992 and 2012 and the season W/L% for the same years. That equation {1.063 + (-.0161*6-year ave)} can be used to approximate future W/L%.

Modeling

What is needed to build a model to do that, however, are parameters yearly recruiting. Under Coach Pelini, his five recruiting classes have averaged 25.3, with a standard deviation of 3.0704. Those two data points, and the regression equation I computed above can be used in a probabilistic model to estimate future W/L%.

The results aren't encouraging.

Based on 1000 iterations, the average recruiting class is (not shockingly) 25.23 (as it should be based on the model). The average estimated W/L% is .58.

The best recruiting class is estimated at 21.51, and the best season W/L% is .68.

The worst estimated recruiting class is estimated at 29.54, and the worst season is estimated at .47.


WhatIf Scenarios

If, going forward, Pelini can average a recruiting class of 20.0, the model predicts that the average W/L% is .72, best is .84, and worst is .61.

If, going forward, Pelini can average a recruiting class of 15.0, the model predicts that the average W/L% is .86, best is .97, and worst is .67.

Conversely, If Pelini averages a recruiting class of 20.0, the model predicts that the average W/L% is .46, best is .56, and worst is .35.

Conclusion

I disagree with Brandon that NU is well-positioned in recruiting. Even if my model understimates W/L% by 10% (roughly 1 game per year), that still means an average W/L% of about .70. I don't think that will be good enough for Husker fans. Pelini has to improve and sustain Nebraska's recruiting significantly if he is going to keep his job.

Which was the Most Impressive 5-Year Dynasty?

CornNation has published my comparisons of the Nebraska and Alabama dynasties in three parts.  Check it out, or download the full PDF.


Thursday, January 3, 2013

Comparing BP and TO's first five seasons

Taking a quantitative look at the TO's and BP's first five seasons, I think it's reasonable to say that both are similar, with TO having outperformed BP in most areas.

Note: all charts are presented with seasons as rows and months as columns, with bowl games and CCGs presented separate from NOV/DEC games. All numbers are averages.

1. First, consider what each had to work with. To do that, I looked at the previous three seasons before the coaching changes.

TO took over a team that had won 2 NCs and had a combined win percentage of .92. BP took over at team that missed a bowl game twice in three years and had a combined win total of .58. Clearly, TO had a head start as a coach.

Click image for larger version. 

Name: Capture1.JPG 
Views: 1 
Size: 43.0 KB 
ID: 4424

2. Next, let's look at each coaches win percentage. Overall, both a very similar, with TO holding a slight edge at .77 to BP's .71. The biggest difference, IMO, is the record in bowl and CCGs. TO had a .80 record in these, BP has an abysmal .29 record. If BP had a similar record in bowl / CCG, he would probably be ahead of TO in win percentage for the first five years.

Click image for larger version. 

Name: Capture2.JPG 
Views: 1 
Size: 54.9 KB 
ID: 4425

3. Now, let's look at the average rank of opponents. For opponents not ranked in the AP Top 20/25, I used the calculated rankings at http://www.jhowell.net/cf/cfindex.htm. These are end of season rankings, so they aren't a perfect proxy, but in the absence of anything else, they serve the purpose.

What jumps out at me is that TO's opponents are, on average, 5-10 points higher ranked. When I look at the teams each played, I believe this is accounted for by the expansion of the season by 2 games...games which are almost always filled with cupcake teams ranked in the 80-125 range and played in the non-conf season. TO played 13 teams ranked >=80. BP played 17 teams ranked >= 80. TO played 6 of those 13 in Aug/Sep, for an average of 1.2 per non-conference season. BP played 11 of the 17 in Aug/Sep, for an average of 2.2 per non-conf season.

BP has an win percentage advantage in the non-conf. This is a direct result of the difficulty of expanded number of low ranked teams played in the non-conf. What is significant in this chart is the average rank of bowl/CCG opponents. TO's opponent's ave rank was 11, BP's was 19. And yet, TO's win percentage is .80 in these games and BP's is .29. TO has a clear performance advantage here, but it should be considered in the light of point #1 - what each coach had to work with at the start of the five years.

Click image for larger version. 

Name: Capture3.JPG 
Views: 1 
Size: 48.6 KB 
ID: 4426

4. Fourth, consider the average MOV in each loss. For TO, I've included ties in the average MOV calculation.

BP's average MOV is much worse in Oct and in bowl/CCGs. Also, the MOV in BP's losses in three of the five seasons were by -20, -20, and -21 points. It is a serious concern when doesn't just get beat, but an average, gets blown out by 3 TDs. IMO, this is the most significant insight of this analysis. It may be the fatal flaw in BP's tenure at NU as well.

It's worth noting, though, that TO's losses in Nov/Dec were pretty bad as well. Three of the five years he had average MOV in those losses of -27, -25, and -31. I don't remember much (okay, anything) about these seasons), but I'm sure these blowouts didn't sit well with NU fans.

Click image for larger version. 

Name: Capture4.JPG 
Views: 1 
Size: 47.6 KB 
ID: 4427

5. Lastly, consider the average MOV in wins.

The average MOV in the non-conf is identical, but the ave MOV in Nov/Dec is 25 for TO and 12 for BP. Overall, the ave MOV is pretty close...24 for TO and 20 for BP.

Click image for larger version. 

Name: Capture5.JPG 
Views: 1 
Size: 47.9 KB 
ID: 4428

6. In conclusion, the results indicate that TO outperformed BP in almost all areas for the first five years. His ave opponent difficulty was greater than BP's and his win percentage against better teams is much better than BP's. Still, I think that the starting point for TO greatly impacted this. The only want to know for certain is to do this again in five years.

IMO, the numbers indicate that BP has done more than enough to make any talk of firing him utter fantasy...for at least a couple of years.