Advice

Predicting 2016 QB Performance Was Really Hard

Thanks to the RotoViz Screener, we can quickly access data all the way back to 2000 and use the data to answer pressing questions. One thing I noticed is that predicting QB performance, even over the course of a full season, is really freaking hard.

First let’s start with why this is important. ADP is largely driven by prior year results. Take a look at 2016 QB ADP plotted against 2015 QB PPR points per game for all QBs who had at least 12 fantasy points per game last year and six games played.1

2017-01-10

Over 63 percent of ADP can be explained simply by prior season performance. On the surface, ADP doesn’t appear to do a terrible job of predicting performance.

2017-01-10-4

An R-squared of 0.45. Not bad, right? Well, what about that big gap in ADP between 136 and 155? Surely we don’t want any of those QBs after that gap, do we? I mean, in 2QB leagues, sure, but in most redraft leagues you probably don’t want to be relying on QB21+ as your top QB. Instead, let’s look at the 20 QBs taken at ADP 136 and earlier and re-draw the graph above.

2017-01-10-5

Well shit. Now our R-squared is only 0.08 meaning only eight percent of the fantasy output per game could be explained by ADP for the top 20 QBs. What’s more, the best-fit line shows that the difference between the earliest and latest QBs in the top 136 in overall ADP was a whopping 2.5 points per game! I dunno about you, but I’d rather just take two QBs between ADP 100 and 136 and make up those 2.5 points (and more) with more bullets at RB, WR, and even TE. Oh right, there’s a name for that draft strategy — Late Round Quarterback, or LRQB.

Maybe we can do better. With the original subset of QBs, let’s correlate 2015 FPPG with 2016 FPPG.

2017-01-10-2

Welp. Pretty gross. But let’s take it one step further, and trim this set of QBs down to just the QBs that played in at least half of their team’s games in both 2015 and 2016.

2017-01-10-3

Even worse. A total mess. For 2016, fantasy relevant QB performance was nearly impossible to gauge based on both ADP and prior year FPPG.

If we look at MFL10 results from 2016, we see this trend in action. Only Drew Brees was on more than an average number of winning rosters, while only Cam Newton had a win rate less than 8 percent among top-20 QBs in ADP.2 You could essentially pick any three QBs from ADP QB2 to QB20 and not be left behind.

Was 2016 an Anomaly?

So far we’ve only looked at 2016, but what if 2016 was an odd-ball year in a sea of otherwise predictable QB performances. To look at this question, I used the RotoViz Screener to query all seasons since 2000 to determine if 2016 was an anomaly. I looked at fantasy relevant QBs that had enough data to make predictions worthwhile. I defined it as a QB that played in half his team’s games in year N and year N+1, while attempting at least 25 pass attempts per game. The single worst QB season that met this criteria was Vinny Testaverde‘s 2001 season, at 10.8 points per game.

Year R-Sq Year R-Sq
2016 0.0657 2008 0.1049
2015 0.022 2007 0.1263
2014 0.1787 2006 0.5474
2013 0.4612 2005 0.26
2012 0.6795 2004 0.3497
2011 0.3293 2003 0.2661
2010 0.2239 2002 0.4077
2009 0.4226 2001 0.3599

Oddly enough, 2016 was the second least predictable year among usable QBs over the last 16 years. Last year takes the cake as the most unpredictable year, and at the time I thought that was an anomaly. But maybe we’re in a new normal? The last three years are among the five least predictable years at the QB position, although the two years immediately prior were two of the three most predictable years.

Can We Do Better?

Even if this is a new normal, I’d like to think we can do better than rely on ADP and prior year FPPG to make accurate predictions at the QB position. But we won’t know until we try. RotoViz’s own projections used a wisdom of the crowds (WOTC) based approach. All of our staff members created projections, and then we averaged the individual projections into a staff composite. Combined, it led to a projection that was slightly more accurate than the trimmed down ADP and FPPG methods above, with an R-squared of 0.14 using the trimmed down data set.

I also created a machine learning model to predict QB performance based on QB clusters that group QBs into similar types.

2017-01-10-6

It certainly performed better than prior year FPPG, the trimmed ADP results, and the trimmed staff composite projections, all of which use some semblance of “reliable QBs” as their data set of QBs. It didn’t perform quite as well as the full ADP data set all the way at top, but it looks more well distributed across the range of QBs than the full ADP data set.

The biggest problem with my QB projections was I didn’t think to incorporate WOTC, which is essentially what ADP is. I did use ADP for my WR projections, and as you’ll see in a future article, it worked quite well. Here were the inputs to my QB model:

  • NFL Draft position (DRAFT)
  • Age (AGE)
  • PPR points in year N (PPR)
  • Interception Rate (paINTRT)
  • Touchdown Rate (paTDRT)
  • Passing Average Margin (paAVGMGN)
  • Rushing attempt market share (ruATTMS)
  • Rushing fantasy points over expectation per attempt (ruFPOEPA)

I think having a full offseason to play with a lot of Josh Hermsmeyer’s air yards work, adding ADP, and trying to find one or two other nuggets of data (I have some ideas) will hopefully help push my QB model into a new level of accuracy. I’d love to hear in the comments section of any other ideas you guys have for improving the accuracy of QB projections.

Subscribe for a constant stream of league-beating articles available only with a Premium Pass.

  1. I removed Tony Romo and Teddy Bridgewater for obvious reasons.  (back)
  2. Again, excluding Romo.  (back)
By RotoDoc | @RotoDoc | Archive

Comments   Add comment

  1. Anything that pushes the limits to improve accuracy is very welcomed by this fantasy player. I almost feel guilty keeping this site from my league mates.
    One article I'd like to see is an explanation of how season long projections are determined. Obviously, you use the projection machine, get input from lots of different writers, but does everyone also use your data from you machine learning? Is there a process of tools that everyone uses or is it left to the individual to decide which tools are best to use?

  2. Great question. We leave it to every writer to decide his or her own process for using the projection machine. That's actually the best way to do it, in my opinion, because it gives us a true wisdom of the crowds approach. It's like the ox story...individual guesses for 787 villagers may have ranged anywhere from 100 to 10,000 pounds, but the crowd average was really close to the actual weight. We want our writers to do the same, using whatever process they think is best.

Discuss this article on the RotoViz Forums