Predicting 2016 QB Performance Was Really Hard
Thanks to the RotoViz Screener, we can quickly access data all the way back to 2000 and use the data to answer pressing questions. One thing I noticed is that predicting QB performance, even over the course of a full season, is really freaking hard.
First let’s start with why this is important. ADP is largely driven by prior year results. Take a look at 2016 QB ADP plotted against 2015 QB PPR points per game for all QBs who had at least 12 fantasy points per game last year and six games played.1
Over 63 percent of ADP can be explained simply by prior season performance. On the surface, ADP doesn’t appear to do a terrible job of predicting performance.
An R-squared of 0.45. Not bad, right? Well, what about that big gap in ADP between 136 and 155? Surely we don’t want any of those QBs after that gap, do we? I mean, in 2QB leagues, sure, but in most redraft leagues you probably don’t want to be relying on QB21+ as your top QB. Instead, let’s look at the 20 QBs taken at ADP 136 and earlier and re-draw the graph above.
Well shit. Now our R-squared is only 0.08 meaning only eight percent of the fantasy output per game could be explained by ADP for the top 20 QBs. What’s more, the best-fit line shows that the difference between the earliest and latest QBs in the top 136 in overall ADP was a whopping 2.5 points per game! I dunno about you, but I’d rather just take two QBs between ADP 100 and 136 and make up those 2.5 points (and more) with more bullets at RB, WR, and even TE. Oh right, there’s a name for that draft strategy — Late Round Quarterback, or LRQB.
Maybe we can do better. With the original subset of QBs, let’s correlate 2015 FPPG with 2016 FPPG.
Welp. Pretty gross. But let’s take it one step further, and trim this set of QBs down to just the QBs that played in at least half of their team’s games in both 2015 and 2016.
Even worse. A total mess. For 2016, fantasy relevant QB performance was nearly impossible to gauge based on both ADP and prior year FPPG.
If we look at MFL10 results from 2016, we see this trend in action. Only Drew Brees was on more than an average number of winning rosters, while only Cam Newton had a win rate less than 8 percent among top-20 QBs in ADP.2 You could essentially pick any three QBs from ADP QB2 to QB20 and not be left behind.
Was 2016 an Anomaly?
So far we’ve only looked at 2016, but what if 2016 was an odd-ball year in a sea of otherwise predictable QB performances. To look at this question, I used the RotoViz Screener to query all seasons since 2000 to determine if 2016 was an anomaly. I looked at fantasy relevant QBs that had enough data to make predictions worthwhile. I defined it as a QB that played in half his team’s games in year N and year N+1, while attempting at least 25 pass attempts per game. The single worst QB season that met this criteria was Vinny Testaverde‘s 2001 season, at 10.8 points per game.
Oddly enough, 2016 was the second least predictable year among usable QBs over the last 16 years. Last year takes the cake as the most unpredictable year, and at the time I thought that was an anomaly. But maybe we’re in a new normal? The last three years are among the five least predictable years at the QB position, although the two years immediately prior were two of the three most predictable years.
Can We Do Better?
Even if this is a new normal, I’d like to think we can do better than rely on ADP and prior year FPPG to make accurate predictions at the QB position. But we won’t know until we try. RotoViz’s own projections used a wisdom of the crowds (WOTC) based approach. All of our staff members created projections, and then we averaged the individual projections into a staff composite. Combined, it led to a projection that was slightly more accurate than the trimmed down ADP and FPPG methods above, with an R-squared of 0.14 using the trimmed down data set.
It certainly performed better than prior year FPPG, the trimmed ADP results, and the trimmed staff composite projections, all of which use some semblance of “reliable QBs” as their data set of QBs. It didn’t perform quite as well as the full ADP data set all the way at top, but it looks more well distributed across the range of QBs than the full ADP data set.
The biggest problem with my QB projections was I didn’t think to incorporate WOTC, which is essentially what ADP is. I did use ADP for my WR projections, and as you’ll see in a future article, it worked quite well. Here were the inputs to my QB model:
- NFL Draft position (DRAFT)
- Age (AGE)
- PPR points in year N (PPR)
- Interception Rate (paINTRT)
- Touchdown Rate (paTDRT)
- Passing Average Margin (paAVGMGN)
- Rushing attempt market share (ruATTMS)
- Rushing fantasy points over expectation per attempt (ruFPOEPA)
I think having a full offseason to play with a lot of Josh Hermsmeyer’s air yards work, adding ADP, and trying to find one or two other nuggets of data (I have some ideas) will hopefully help push my QB model into a new level of accuracy. I’d love to hear in the comments section of any other ideas you guys have for improving the accuracy of QB projections.