Using Analytics to Project Carson Wentz and Jared Goff
Recently RotoViz debutante Chris Hatcher wrote an interesting piece about quarterback ball velocity and how that translates into one metric of passing success at the NFL level. I decided to take his work a step further and used logistic regression to model QB success.
There are multiple ways to define QB success. One might be with fantasy stats. Another might be raw totals. The metric Chris used was whether or not a QB threw for an AYA of 7.0 or higher for at least one season in his career, while also starting 8+ games in said season. I will also use this as my criteria for success. I like this criteria for a few reasons:
- The data was easy to get1
- It leaves out rushing statistics, so we’re focusing only on passing
- It incorporates touchdowns and interceptions along with yards
- It strongly correlates with NFL win percent
- It doesn’t matter how long a QB plays, if they reach the threshold, they usually do so early in their careers. This is important because you might expect the criteria to bias against players with only one or two seasons under their belt. But both Marcus Mariota and Jameis Winston met the threshold in their first year. I tested number of seasons in the NFL against probability of success, and found there was no correlation (p=0.8). This held true even when controlling for other variables, such as draft position.
The data set I used for ball velocity is the same data set Chris used – combine data on every thrower since 2008 as recorded by Ben Allbright. I then grabbed other combine results and player stats to use when training and testing my model. In the few cases where combine data wasn’t available, I used pro-day numbers and on one occasion (for John David Booty) I imputed his 3-cone time from other metrics.
In all, there are 94 quarterbacks from 2008-2015 with velocity data. Of these 94 QBs, 73 played at least one snap at the NFL level (77.7 percent). Breaking that down further, only 15 of the 73 QBs that played met the success criteria that Chris established in his article. In other words, most QBs fail.
There are plenty of quarterbacks who either didn’t throw at the combine, or at least didn’t have the ball velocity stat recorded. As such, this analysis only applies to QBs who had a ball velocity recorded.
Building the Model
I held back a random sample of 20 players, thus training my model on 74 of the 94 players. This left me with four players in the test set of 20 who met the threshold. The reason for holding back a random sample rather than building the model on, say, 2008-2013 data and testing on 2014-2015 is to eliminate any potential bias that might be introduced by looking only at the older data. As we know, the NFL has trended toward a passing league especially with pass catching running backs, so I wanted to account for that.
When training my model, I chose to use the Bayesian Information Criterion (BIC) to avoid overfitting to the training set. In doing so, I found the model with the lowest BIC2 included the following paramters:
In other words, draft position3 was the most statistically significant parameter when predicting this metric of success. Final year AYA was next, in essentially a dead heat with velocity. Interestingly, the 3-cone drill time was borderline significant, but helped the predictive power of the model so it was left in. However, this isn’t all that surprising, because rushing QBs tend to have a better AYA even when controlling for other variables. Since the 3-cone, shuttle, and 40-yard dash are all positively correlated with each other, we can make an educated guess that the 3-cone represents some aspect of rushing ability.
Because there was missing data in some of these factors, this left me with 68 players in the training set and 17 players in the test set.
The model did very well against data it was built with, placing 63 of the 68 players (92.6 percent) in the correct category.
However, this is a retrodictive look. A better test comes by making predictions on withheld data.
When I tested the model on the held back set, it correctly predicted 13 misses out of 13, and three successes out of four, for a total of 16 of 17 correct predictions (94.1 percent).
This validated these parameters, so I rebuilt the model on the full data set using the same four variables to have a more robust model. You’ll notice the p-values improved across the board…more data helps! Surprisingly, velocity and 3-cone became more significant than final season AYA for the model built on the full data set. Perhaps the velocity piece shouldn’t come as a surprise. After all, Peyton Manning put up the worst AYA of his career (5.0) in his final NFL season with a shot arm.
The final model only missed on these six players:
Had this model been in use and developed back in 2011, it could have told the Minnesota Vikings to avoid spending the 12th overall pick on Christian Ponder. You can find the full results at bottom.
So how does the model apply to the 2016 class? Here are their numbers:
Carson Wentz actually leads Jared Goff by the slimmest of margins, but both are almost mortal locks to meet the success criteria. Paxton Lynch is the only other QB favored to do so from this class.
It’s interesting to see Vernon Adams at fourth most likely, right in line with Justin Winn’s post-combine QB rankings, despite going undrafted and overlooked. His 3-cone time really helped, along with an AYA of 11.2 in his only college season. Perhaps I need to bake in a college experience factor and see if that has any predictive power. That could temper expectations for Adams if so.
Connor Cook came in fourth in the final RotoViz Scouting Index, then was selected by the Oakland Raiders at pick 100. Oops. He shows up dead last among those in the 2016 class where all the data was available. His ball velocity of only 50 MPH really hurts his chances. Add in questions about his leadership, and he’s a guy to avoid like the plague.
I expect more variables to pop up as significant as the data set becomes larger with time. Age, for example, was significant but also pushed into overfitting territory. Weight also showed up as significant, with lighter QBs actually faring better. This is likely because it is correlated with the rushing prowess of a QB, so I didn’t want to bake in an extra model term because that would introduce an unacceptable level of multicollinearity beyond what already exists.
|John Parker Wilson||2009||6.5||7.53||330||5.8||58||0.13%|
|John David Booty||2008||7||7.79||137||4.9||51||0.00%|