The 2017 QB Prospect Model: Draft Deshaun Watson at Your Own Peril

Last year I created a model to predict quarterback success at the NFL level for prospects. The model was useful in that it helped identify traits that transfer to the NFL level, but it was certainly overfit and didn’t bring in enough data. I’ve updated the model for 2017, and the improvements paint a much better story of each QB’s NFL prospects. The results will probably surprise you.

Like last year, I’m going to define success by a QB throwing for an AYA of 7.0 or higher for at least one season in his career, while also starting 8-plus games in that season.

To refresh your memories, this criteria is useful for the following reasons:

  1. The data was easy to get.
  2. It leaves out rushing statistics, so we’re focusing only on passing success.
  3. AYA incorporates touchdowns and interceptions along with yards, which are all fantasy relevant stats.
  4. And perhaps most importantly, it strongly correlates with NFL win percentage.

The data set I used for ball velocity comes from Ben Allbright’s combine data on every thrower since 2008. I then grabbed other combine results and player stats to use when training and testing my model. In the few cases where combine data wasn’t available, I used pro-day numbers. This gave me a data set of 103 full records with no missing values.

There are plenty of quarterbacks where the data is incomplete that I am not using for this model’s purposes. Quite often, it’s because a QB didn’t throw at the combine, or at least didn’t have a ball velocity recorded, which is unfortunate because it’s predictive of NFL success. For example, I don’t have ball velocity data on Matt Ryan, so he was not used in building the model.

In the end, that’s okay, because there are six QBs in the 2017 class, including the big four, that have complete sets of data, so we’re still comparing apples to apples.

Building the Model

To build the model, I took the complete data set, which included the following data: height, weight, age, 3-cone, shuttle, hand size, throwing velocity, and a film grade, and added on final college year AYA, draft position,1 and a bin for the number of years out of college.

I’ll spare you the details, but the most statistically significant split for the out of college bin was between four and five years removed from college. In other words, players who have completed four years post-college fall in one bin, and players who are going into their first through fourth years out of college fall in the other bin. This is important because it’s certainly much easier to hit the criteria of one year of a 7.0 AYA if you’ve played several years, compared to having only one or two years of playing.

For the 2017 QB class, I imputed their draft position by using the RotoViz Scouting Index and comparing that to where past prospects were drafted. That gave me the following average picks:

QB Avg.Dpos
DeShaun Watson 5.6
Mitch Trubisky 15.5
DeShone Kizer 20.4
Pat Mahomes 68.6
Davis Webb 167.9
Nathan Peterman 189.0

From there, I took all these data points, and ran them through a feature selection process — which is a fancy name for finding the best predictors for a model. Here were the results of that process:


You probably have no idea what you’re looking at. This is just 15 different models with different combinations of predictors, with the best model listed at the top.2 If the box is filled in, that means the variable was used in the model. So the best model overall had years out of college (ExpBin), hand size (Hand), throwing velocity (Vel), logarithm of draft position (Log.D.Pos), final college year AYA (Last.AYA), and the film grade (Film) as inputs.

I validated the top model candidates against a randomly withheld test data set, and sure enough, the same six variables from the feature selection process also gave the best out-of-sample results in the test set with multiple metrics for accuracy.3 I did this with multiple randomly withheld subsets, and almost every single time the same six variables gave the best out-of-sample results, ensuring this was the best model to use.

From there, I rebuilt the model without holding back any data, except the data we were trying to predict – the 2017 draft prospects, along with those prospects that hadn’t hit the required four years out of college, because it’s still possible in their remaining eligibility they hit the success criteria at least once.

It’s interesting to note that the 3-cone did not pop up as helpful, like it did last year. There’s probably two reasons for that. First, I’m using a more sophisticated model. Next, it’s also possibly baked into some of the other metrics, like velocity, draft position, and film grades.

Model Results

First, let’s look at the model results for all QBs that have been out of college for at least one year and have a complete set of input factors. As mentioned, this model takes into account whether they’ve also been out of college at least four years into their model-predicted probabilities of having achieved success yet.

Cam Newton98.3%Y
Mark Sanchez96.5%Y
Josh Freeman91.3%Y
Russell Wilson88.5%Y
Joe Flacco83.5%Y
Andy Dalton82.4%Y
Marcus Mariota79.7%Y
Carson Wentz74.8%N
Nick Foles70.7%Y
Kirk Cousins68.6%Y
Jameis Winston66.0%Y
Paxton Lynch65.3%N
Blake Bortles62.5%Y
Teddy Bridgewater53.0%N
Cody Kessler37.9%Y
Brian Brohm37.8%N
Brandon Weeden37.0%N
E.J. Manuel35.8%N
Colin Kaepernick32.6%Y
Jared Goff27.8%N
Brett Hundley22.3%N
Chad Henne17.8%N
Dak Prescott17.1%Y
Bryce Petty16.9%N
Jimmy Garoppolo16.6%N
Ryan Nassib15.5%N
Christian Ponder15.1%N
Jake Locker14.8%N
Pat Devlin14.5%N
Scott Tolzien14.4%N
Paul Smith12.8%N
Ricky Stanzi11.9%N
Tyrod Taylor11.5%Y
Tyler Wilson11.4%N
Austin Davis11.0%N
Logan Thomas9.6%N
Jacoby Brissett9.5%N
Keith Wenning9.1%N
Mike Reilly9.0%N
Jeff Mathews8.9%N
Stephen Morris8.4%N
Connor Cook8.2%N
Christian Hackenberg8.1%N
Levi Brown7.6%N
Kevin Hogan7.4%N
Case Keenum7.1%N
Drew Willy6.6%N
Sean Mannion6.4%N
Tony Pike6.2%N
T.J. Yates6.0%N
Chandler Harnish6.0%N
Brett Smith5.8%N
Mike Glennon5.6%N
Tom Savage5.3%N
Josh Johnson4.9%N
Chase Daniel4.9%N
Pat White4.7%N
Nathan Enderle4.4%N
Nate Sudfeld4.2%N
Tajh Boyd4.2%N
Brandon Allen4.0%N
Ryan Lindley3.9%N
Kellen Moore3.8%N
Rhett Bomar3.6%N
John Parker Wilson3.6%N
John Skelton3.5%N
Graham Harrell3.4%N
Tyler Bray3.3%N
Jevan Snead2.8%N
Jerry Lovelocke2.6%N
Dan LeFevour2.5%N
Curtis Painter2.2%N
David Fales2.2%N
Max Hall2.0%N
Cody Fajardo2.0%N
Jeff Driskel1.9%N
Landry Jones1.8%N
James Vandenberg1.7%N
Erik Ainge1.7%N
Connor Shaw1.7%N
Vernon Adams1.6%N
Brandon Doughty1.5%N
Shane Carden1.4%N
John David Booty1.4%N
Tom Brandstater1.4%N
Sean Canfield1.2%N
Stephen McGee1.2%N
Colby Cameron1.1%N
Matt Flynn1.1%N
Dustin Vaughan0.9%N
Josh Woodrum0.9%N
Brandon Bridge0.9%N
Bryn Renner0.8%N
Tim Hiller0.6%N
Zac Robinson0.6%N
Mike Kafka0.6%N
Matt Scott0.5%N

The misclassification rate is only 7.1 percent, which is fantastic, and intuitively it appears as if the vast majority of the best quarterbacks appear at the top, whereas players like Christian Ponder weren’t predicted to be very successful. It’s even interesting to see Brett Hundley ahead of Ponder.

Because some players haven’t met the four-year threshold, I also included them as “prospects” to find out their probabilities of being successful at any point in their NFL career. This also includes the 2017 crop of QBs, so let’s take a look at those results.

Carson Wentz92.5%
Paxton Lynch82.7%
Mitch Trubisky73.8%
Teddy Bridgewater68.1%
DeShone Kizer43.9%
E.J. Manuel43.5%
Jared Goff38.6%
Pat Mahomes27.4%
Brett Hundley27.3%
Jimmy Garoppolo25.3%
Bryce Petty21.7%
DeShaun Watson20.4%
Ryan Nassib20.0%
Tyler Wilson16.3%
Jacoby Brissett13.7%
Keith Wenning13.6%
Jeff Mathews13.2%
Stephen Morris12.6%
Christian Hackenberg12.1%
Logan Thomas12.0%
Connor Cook11.3%
Kevin Hogan10.6%
Sean Mannion10.4%
Brett Smith8.4%
Tom Savage8.3%
Brandon Allen7.7%
Nathan Peterman7.0%
Nate Sudfeld6.9%
Mike Glennon6.7%
Tajh Boyd6.5%
Tyler Bray6.2%
David Fales3.6%
Jeff Driskel3.6%
Jerry Lovelocke3.5%
James Vandenberg3.4%
Connor Shaw3.3%
Cody Fajardo3.2%
Vernon Adams3.0%
Brandon Doughty3.0%
Landry Jones3.0%
Davis Webb2.9%
Shane Carden2.6%
Josh Woodrum2.4%
Colby Cameron2.4%
Brandon Bridge2.3%
Dustin Vaughan2.2%
Bryn Renner1.6%
Matt Scott1.0%

Here we can see Carson Wentz jumps from having a 74.8 percent probability of at least one successful season in his first four years to a 92.5 percent chance of having at least one successful season at any point in his career. It’s very interesting to see the model only gives Jared Goff a 38.6 percent chance of meeting the success criteria in his career. Further down the list, the Bears new QB Mike Glennon has yet to have a successful year, and the model gives him only a 6.7 percent chance of having one in his career. Ouch! The model doesn’t like Glennon’s poor throwing velocity and his pedestrian final year college AYA of only 6.9. No QB who has thrown under 50 MPH at the combine has gone on to have an NFL season with an AYA of 7.0 or greater.

We can also see the model favors Mitchell Trubisky out of the 2017 QB class. The model likes Trubisky’s 9.1 final year AYA and his film grade of 82, in addition to his mid-first-round valuation. Deshone Kizer comes in second at 43.9 percent probability of success, followed by Pat Mahomes at 27.4 percent, despite an imputed draft position as a late-second or early-third-round pick. If Mahomes gets drafted even higher than that, watch out!

On the flip side, the model is not a big fan of DeShaun Watson. Watson’s ball velocity of 49 and relatively low film grade compared to prior first round prospects gave him a big knock. Watson also has the worst final season AYA of the big-four QBs from this class. If he falls to the middle or end of the first round, that 20.4 percent success probability will drop even further. Watson of course still adds fantasy value with his legs, but don’t get too excited about him as a passer.

I’ve added all the QB data below, so you can build your own models or play with the data as you see fit!

Cam Newton2011611.26.92105624860504.189.8752190Y11
Marcus Mariota2015211.56.8720.75622260364.119.8752189Y10
Carson Wentz201618.76.8620.75723760524.15102384N00
Jared Goff201619.47.17105821560414.4792180N00
Blake Bortles201439.67.0831.15623260504.219.3752282Y10
Mark Sanchez200989.47.0651.65722760214.2110.52284Y11
Colin Kaepernick201168.66.85363.65923360454.189.1252374Y11
Jameis Winston201527.77.16105523160364.369.3752190Y10
Paxton Lynch201619.47.14263.35924460554.2610.252282N00
Joe Flacco200898.66.82182.95523660634.279.6252385Y11
Andy Dalton201169.96.93353.65621560204.279.52383Y11
Teddy Bridgewater2014310.37.17323.55821460214.29.252194N00
Jake Locker201166.66.7782.15423160244.129.6252278N01
Mike Reilly2009810.16.763305.85821460304.11102476N01
Russell Wilson2012511.86.97754.35520751054.0910.252392Y11
Josh Freeman200987.87.11172.85724860564.43102185Y11
Jimmy Garoppolo20143107.04624.15622660204.269.1252271N00
Brandon Weeden201258.67.36223.15922160344.459.6252872N01
Chandler Harnish201259.16.782535.55721960154.159.252369N01
Kirk Cousins2012587.051024.65921460254.59.8252385Y11
E.J. Manuel201348.87.08162.85423760454.2110.3752380N00
Austin Davis201257.66.733305.85821960154.1110.3752271N01
Logan Thomas201436.57.051204.86024860604.1810.8752275N00
Case Keenum2012510.66.873305.85520860054.289.1252470N01
Christian Ponder2011676.85122.55122960204.0910.252370N01
Pat Devlin2011611.57.083305.85622560334.329.8752365N01
Vernon Adams2016111.26.823305.85320051124.29.1252366N00
Christian Hackenberg201617.27.04513.95622360354.3392172N00
Nick Foles201257.67.14884.55824360504.6810.6252384Y11
Bryce Petty201529.66.911034.65323060304.13102380N00
Scott Tolzien201169.46.843305.85521260204.12102367N01
Paul Smith200899.57.023305.85720860114.299.752379N01
Josh Woodrum201617.36.743305.85623160274.319.252360N00
Kevin Hogan20161106.91625.15321760304.3110.252367N00
Keith Wenning201439.17.071945.35621860304.26102370N00
Brandon Allen201619.97.062015.35521260144.338.6252370N00
Tyler Bray201348.37.23305.85923260614.519.252173N00
Matt Scott201347.16.693305.85421560213.999.1252266N00
Brett Hundley201528.66.9314755322660303.9810.52184N00
Brian Brohm200898.67.135645323060274.559.752295N01
Jacoby Brissett201617.17.17914.55623660344.539.752371N00
Tyrod Taylor201169.56.781805.25021760064.09102169Y11
Levi Brown201078.57.073305.85622960344.439.52460N01
Dak Prescott201618.77.111354.95423060144.3210.8752272Y10
Brett Smith201437.46.983305.85620760164.19102179N00
Drew Willy200987.67.183305.85821560314.459.52276N01
Sean Mannion201526.97.29894.55722960604.3992369N00
Stephen Morris201438.57.363305.85921360204.4910.1252266N00
Rhett Bomar200985.76.9115155522560224.0692377N01
Pat White200987.17.06443.85219760024.429.252365N01
Tyler Wilson2013487.221124.75521560214.398.6252381N00
Cody Kessler201618.57.32934.55522460124.0110.8752376Y10
Tom Savage201437.67.331354.95722860404.369.6252470N00
Ryan Nassib201348.17.341104.75622760214.5310.1252374N00
Curtis Painter200985.772015.35622560274.429.252370N01
Ricky Stanzi201169.46.951354.95022360434.43102371N01
James Vandenberg201345.26.953305.85722660264.529.752363N00
Jeff Mathews201437.87.143305.85622560334.3610.382270N00
Landry Jones201347.97.121154.75322560414.39.1252470N00
Chase Daniel200988.27.283305.85721860004.319.252269N01
Chad Henne200896.77.175745323060274.492286N01
Tajh Boyd201439.87.332135.45422260104.239.6252368N00
Cody Fajardo201525.86.953305.85522360104.19.52366N00
Brandon Bridge201526.17.183305.85722960404.379.252364N00
Jeff Driskel201619.47.192075.35223460354.259.752368N00
T.J. Yates2011686.963305.85221960334.1210.252368N01
Tom Brandstater200986.76.933305.85322060504.379.52478N01
Nathan Enderle201166.37.131605.15424060414.469.8752370N01
Max Hall201078.87.073305.85220960054.3592465N01
Colby Cameron201348.76.983305.85121260214.289.1252366N00
Nate Sudfeld201619.27.421875.25423260324.489.6252265N00
Connor Shaw2014310.17.073305.85020660004.339.252264N00
Dan LeFevour201078.16.931815.24923060324.229.252371N01
Dustin Vaughan201439.17.253305.85323560504.438.8752361N00
Brandon Doughty2016110.47.492235.45321060224.529.1252465N00
Bryn Renner201437.57.223305.85422860304.369.1252469N00
Mike Kafka201076.56.963305.85222560314.379.252267N01
Connor Cook201618.17.211004.65021760304.289.752379N00
Tony Pike201078.47.062045.34922360564.53102470N01
John Parker Wilson200986.57.533305.85821960144.599.52372N01
Josh Johnson2008912.67.561605.14921360264.4292172N01
Kellen Moore201259.77.413305.85219760004.569.52262N01
Shane Carden201527.97.173305.85121860204.459.752360N00
Jevan Snead201076.17.083305.85221960304.33102265N01
David Fales201438.87.551835.25321260204.59.252372N00
John Skelton201078.67.173305.85024360534.339.752270N01
Tim Hiller201076.17.13305.85222960404.549.1252360N01
Sean Canfield201077.67.262395.55122360364.399.252361N01
Graham Harrell2009897.453305.85223360214.569.52373N01
Stephen McGee200986.97.343305.85322560274.4992380N01
Zac Robinson201076.17.242505.55221460244.492363N01
Matt Flynn200896.57.212095.35023160224.349.252277N01
Erik Ainge200897.17.511625.15222560544.69.1252180N01
Ryan Lindley201257.37.521855.25222960364.45102271N01
Mike Glennon201346.97.49734.34922560714.529.52373N00
Jerry Lovelocke201526.87.473305.85124860524.5110.52261N00
John David Booty2008977.791374.95121860234.588.52370N01
Mitch Trubisky201709.16.8715.52.75522260214.259.52282
Pat Mahomes201708.56.8868.64.26022560204.089.252179
DeShaun Watson2017086.955.61.74922160244.319.752179
DeShone Kizer201708.17.420.43.05623360424.539.8752279
Nathan Peterman201709.37.14189.05.25322660214.319.8252274
Davis Webb201706.96.92167.95.15922960504.219.252270

Subscribe for a constant stream of league-beating articles available only with a Premium Pass.

  1. I actually used the logarithm of draft position  (back)
  2. According to this particular feature selection algorithm and Cp statistic.  (back)
  3. Including AUC and misclassification rates.  (back)
By RotoDoc | @RotoDoc | Archive

Comments   Add comment