The Scouting Model vs. The Algorithm
This piece was originally published on FantasyDouche.com in spring of 2012 and addresses fantasy scoring only – although fantasy scoring is actually a pretty good proxy for our collective impression of what makes a good wide receiver. To put it another way, try to think of a successful real world receiver who isn’t a successful fantasy receiver. This piece pits a model that includes simple box score stats and also Market Share of college team yards vs. Draft Position in a game of “Who can pick the better WR?”
Also, while Percy Harvin’s 2012 season will now make some of the examples in this piece seem off, Harvin played the majority of his rookie contract at a level that would still make these conclusions valid. Basically, teams should not be drafting WRs hoping that in their 4th pro season they finally start outproducing WRs picked after them.
I sort of assume that some NFL teams use various forms of analytics in their draft process. I would be really shocked to find out that the Packers don’t use some form of analytics. The way that they won a Super Bowl in a year that they were decimated by injury was the first red flag for me that they are doing something a little different. Both Greg Jennings and Jordy Nelson are receivers that a statistical model would have liked very much coming out of college, and yet the Packers got both receivers in the 2nd round.
But I also assume that most teams are sort of blissfully unaware that analytics might provide any value. I’ve discussed in the past that the league’s phrase “Stats are for Losers” puts it in sort of a weird anti-science place. It’s also common to hear people say “Well, I don’t think you could use stats for that. It’s too complex.” (every time I hear someone say that, I think “Oh, because you’ve probably tried right?”).
One thing I’ve talked about is how NFL teams don’t optimize the way that they select receivers. While NFL draft position does have some explanatory power over receiver production, you also have to consider that the system is rigged in favor of the draft evaluators. The same coaching staff/front office that selects a player is also the same group of people that assign playing time. If there are mistakes in the draft evaluation, they are going to carry forward. So really the explanatory power that draft position has on receiver production isn’t very impressive.
One thing I did recently to try to account for the inherent bias in the system that favors draft evaluators was to look at the draft in terms of just wide receivers that had been picked right after each other.
To think about what I did in simple terms, imagine that I created a game that was going to pit an NFL draft evaluator (except we’ll use draft order as a proxy for the opinion of an evaluator) against a formula. The evaluator and the formula will be presented with two wide receivers and will have to try to guess which will have more production in the early part of their career. Each set of two wide receivers will only be guys who had been picked after each other.
For example, in the 2009 draft, the game would go like this:
Heyward-Bey vs. Crabtree
(evaluator picks DHB, algorithm picks Crabtree, algorithm wins)
Crabtree vs. Maclin
(both the evaluator and the algorithm pick Maclin)
Maclin vs. Harvin
(both pick Maclin)
Harvin vs. Nicks
(evaluator picks Harvin, algorithm picks Nicks, algorithm wins)
The evaluator would make its guess and the algorithm would rely on its formula for its guess. Then when the evaluator and the formula disagreed, we would tally up which was right and which was wrong.
But again, for our purposes we’ll just use draft order as a stand in for the evaluator. So can draft order beat a simple algorithm at a game of prediction when choosing between two wide receivers? It turns out it can’t. The algorithm would win.
Out of the 120 receiver pairs that I looked at over the last decade, my simple algorithm disagreed with the draft order in 49 instances. Out of those 49 disagreements over which player should be taken, the algorithm was right almost 60% of the time, compared with 40% for the draft order.
But the interesting thing is that when the algorithm disagreed with draft order and the algorithm was right, it was right by a larger margin than the draft order was right by when it won. The following graph shows this difference. When the algorithm was right, the receiver that the algorithm picked outperformed the receiver that the draft order picked by 2.84 fantasy points per game. But when draft order was right, the receiver that it picked outperformed the algorithm’s receiver by only 2.48 fantasy points/game.
The results are even more impressive for the algorithm if you consider that the algorithm is always taking the lower picked receiver. Even though we are trying to isolate for the effect of draft position, the algorithm still does only get to be right on lower picked receivers, who are also going to have marginally fewer opportunities. The graph below shows that on the algorithms wins, it is winning with a receiver that went off the draft board about 11 spots behind the competing receiver. In the draft board’s 20 wins, it’s winning receiver went off the draft board 12 spots in front of the algorithm’s receiver. Even though we are trying to level the playing field for the algorithm, it is still at a disadvantage when it does win, and an even greater disadvantage when it loses.
Even when we try to level the field in a wide receiver picking contest, an algorithm is still at a disadvantage against the NFL’s current evaluators, and yet the algorithm can still “outpick” the evaluators. This is another in the growing list of reasons why NFL teams should increase their use of analytics.