via tequila hardness

The difficulties associated with intelligence analysis are often attributed to the inadequacy of available information. Thus the US Intelligence Community invests heavily in improved intelligence collection systems while managers of analysis lament the comparatively small sums devoted to enhancing analytical resources, improving analytical methods, or gaining better understanding of the cognitive processes involved in making analytical judgments. This chapter questions the often-implicit assumption that lack of information is the principal obstacle to accurate intelligence judgments.¹

If you think about it, the CIA analyst² and NFL GM are engaged in a similar task: analyzing new information about disparate variables (the athleticism and production of college players), then projecting how the addition of specific units of observation (the players) to a complex system (an NFL team) will affect relationships within a larger ecosystem (the team’s standing within the NFL).

This is a flawed process. If you’re a fan of an NFL team, I’m sure you can identify many player evaluations and draft picks that went awry for you team.³ But can the process be improved?

The Goal: Improved Player Evaluation

There are two elements at work in the process of gathering intelligence and formulating predictions: the information itself, and the model used to interpret the information.⁴ In other words, we filter truths (data) through our beliefs (models) in an attempt to obtain knowledge (good predictions about football players’ future performance). ⁵

Show Me Your Model

If we’re going to improve the player evaluation process, we need to look at both components. Beliefs basically boil down to our conceptual framework. Data goes in, gets processed by our beliefs, and gets spit back out as a decision or prediction. For our purposes, models come in two varieties: Narrative driven or Data driven.

The Implicit Narrative Driven Model

This concept-driven, or “soft” model has natural appeal. It’s rooted in the same narrative structure that underpins not only the human mind, but culture and society as well. ⁶ Our brains compensate for an overwhelming amount of information by creating explanatory stories, or narratives. This is the type of model that many analysts use: observe the players, and interpret what’s seen based on an internal conceptual framework. But these concepts or mental models are not always accurate, nor are they best way to formulate predictions.

The problem is that “the nature of soft systems means that their models are generally incomplete and have other inadequacies.”⁷ Interpreting information using a narrative model that resides in one’s mind is problematic because there’s a really good chance it won’t make sense to someone else. Also, the same data fed into different mental or narrative models will likely produce very different predictions.

Furthermore, there’s a problem of feedback and correction- because the model resides inside myself, it’s very easy to resist, discount, or overlook information that contradicts my model. This is a bad way for a large organization to go about analyzing information.

For examples of this, I refer you to old school NFL scouting, wherein “scouts cannot be wrong because their observations are personal,” even though they “seem to know very little about what type of players are currently playing in the league.”⁸ In other words, their narrative models aren’t keeping up with reality.

Big Brother takes it a step further:

“In the absence of any agreed-upon analytical schema, analysts are left to their own devices. They interpret information with the aid of mental models that are largely implicit rather than explicit…. Analysts are often surprised to learn that what are to them self-evident truths are by no means self-evident to others, or that self-evident truth at one point in time may be commonly regarded as uninformed assumption 10 years later.” ⁹

The Explicit Data Driven Model

Data driven, externalized models have several advantages over narrative models. Model components are explicit, use agreed upon definitions and parameters, and are more easily communicated from one person to another. So “anybody” can apply a data driven model and generate the same, or similar, predictions. Think this might be helpful if you’re trying to get a room full of scouts to evaluate players consistently?

Furthermore, the data driven model exists outside of the analyst, so a criticism of it isn’t a criticism of the analyst.¹⁰ In practical experience, it’s much easier for a group of people to wrestle with a data driven model than a concept driven one.

Likewise, mathematical tools like linear regressions can be used to quantify relationships between outcomes and predictors to determine which were most to credit or blame for the accuracy of a prediction. This addresses the feedback and correction problem experienced in a narrative model: there are “objective standards for judging the quality of analysis, inasmuch as the conclusions follow logically from the application of the agreed-upon model to the available data.” ¹¹

In the world of football, analysts of all stripes are challenged in this regard. The sheer number of variables, the complex interactions between them, and the incomplete nature of the available data pushes us towards using Narrative models. But insofar as we can pull ourselves towards the Data Driven end of things, we’ll have an advantage in that our models will be more accessible, and thus more amenable to post prediction validation and corrective scrutiny.

So, in order to improve player evaluation, using an explicit data-driven model seems like our best course of action. Now, what about the information we feed into that model?

Variably Cloudy

The two types of models I’ve outlined lend themselves to naturally process information differently.

Narrative-based conceptual models are good at “making sense out of” vast amounts of vaguely defined variables. But they’re susceptible to influence by extraneous variables and do a poor job of understanding which variables have the biggest influence on generating predictions. More on this later.

Data driven models however utilize discrete, well defined variables. This is really the key distinction to keep in mind throughout the rest of this article. I’ll explain it this way:

• The goal of player evaluation is to predict which players will be the most successful in the NFL
• Well defined variables¹² make it possible to run controlled data experiments and establish strength of correlation between variables
• Inconsistently defined or differently understood variables do not
• Data driven models utilize (require) well defined variables
• Narrative models do not. They may start with well-defined variables, but are highly prone to definitional slippage
• Data driven models can be cross checked and refined more easily than narrative models
• Therefore data driven model generates better and more consistent predictions in the long run

Too Much Information, Driving Me Insane ¹³ 

The first reaction to “improving predictions” is “get more information”, right? We see this in real life, in both the Intelligence community¹⁴ and the NFL, where teams employ a burgeoning pool of scouts, evaluators, and private investigators.

But is all this information actually helpful? The short answer is no. See the quote at the beginning of this article; the CIA itself has no problem gathering scads of data. But most of it isn’t useful, because they lack the ability to properly analyze it.

Consider this experiment.¹⁵ A group of experienced horse race handicappers was each presented with a set of 88 horse-performance related variables, and asked to identify the 5, 10, 20, and 40 that were most important to their predictive model. Then they were presented with data from real (but unknown to them) horse races. First they were presented data for their top five variables for each horse in the race, and asked to predict the order of finish. Then they were given the data for their top 10 variables and asked to re-predict the order of finish, and so on.

The handicappers’ predictions were just as accurate with only 5 bits of information as they were with 40. More information did not improve the accuracy of their projections. What did change? Their pre-race confidence in their predictions:¹⁶

The cautionary takeaway for NFL teams (and for you, O Fantasy GM): more information can make you more confident, but not necessarily more accurate. At five data points, the accuracy of the handicapper’s model and the belief s/he had in its accuracy were in sync. As the handicapper obtained more data, the accuracy of their predictions stayed the same, but they believed they were getting more accurate. Think this might make an NFL decision maker take an unnecessary risk, or overestimate their own prowess?

The CIA sums this up nicely:

“Once an experienced analyst has the minimum information necessary to make an informed judgment, obtaining additional information generally does not improve the accuracy of his or her estimates. Additional information does, however, lead the analyst to become more confident in the judgment, to the point of overconfidence.” ¹⁷

Do I Know You?

So why do handicappers desire more information if it doesn’t improve accuracy? The CIA explains:

“Experienced analysts have an imperfect understanding of what information they actually use in making judgments. They are unaware of the extent to which their judgments are determined by a few dominant factors, rather than by the systematic integration of all available information. Analysts actually use much less of the available information than they think they do.”¹⁸

So analysts are taking in lots of data, but don’t realize they’re only using a small part of it. Worse, they don’t realize which data elements they’re actually using. Can this issue be corrected?

No. Thanks for reading.

Just kidding. Of course it can. The main point is that beyond a certain point additional information increases confidence¹⁹ but not accuracy. However, that doesn’t mean it’s not possible to make good predictions.

The secondary point is that narrative driven models fare poorly in terms of understanding which information they’re actually using. Another experiment illustrates this principle. A group of stockbrokers was asked to predict the future performance of a basket of securities. Afterwards they were asked to explain which variables (P/E earnings, volume, support/resistance, etc.) they used to make their prediction. Their recollection of which variables they used, and which ones they prioritized, differed drastically from what they actually used. ²⁰

So what’s an analyst²¹ to do? Before running out and watching film cut ups of every prospect in the nation, figure out what type of information to gather, and how much of it. In the context of football, this question could provide enough content for a website. ²² But let’s answer the question more universally.

Decision Making in the Absence of All Information

A prevailing theory of intelligence work – and NFL player evaluation – is that more information is better. We’ve already demonstrated that’s not the case, but when using a narrative based model, there’s an additional danger to too much information: building mosaics.²³

Or rather, a mosaic theory of analysis, wherein every available scrap of information is gathered and assembled to see if it “tells a story that makes sense.” This is one of the primary shortcomings of narrative based models: this process is “more art than science.”²⁴This approach purports that each bit of data (a) has value, (b) is like a puzzle piece, and that (c) some clever analyst will assemble the pieces into a coherent picture, or mosaic.²⁵

Neat theory, but it isn’t true. Our brains are more likely to work backwards, starting out with a picture in mind, and then finding the puzzle pieces to fit it.²⁶ So gathering every bit of available information isn’t useful in a narrative approach to analysis.²⁷ With a data driven model, it’s much easier, but not always necessary, to accommodate large volumes of data: once the model is built, it’s very easy to pour new information into it. More on this in a bit.

The Doctor Will See You

What’s a better method of data collection? One example comes from the medical field, where it’s been found that there’s little correlation between data collection and accuracy of diagnosis. In fact, clinicians can get stuck collecting data, and either fail to move to the diagnostic phase, or make diagnoses that are suboptimal or incorrect. Conversely, physicians who focus on generating and testing hypotheses produced superior (and quicker) diagnoses despite gathering less information overall.²⁸ This should sound familiar; remember this earlier quote from the CIA? “Once an experienced analyst has the minimum information necessary to make an informed judgment, obtaining additional information generally does not improve the accuracy of his or her estimates.”

Predictive analysis should be approached similarly: Observe symptoms, formulate a hypothesis, and then test it. If the hypothesis is supported, a diagnosis (prediction) is offered. If not, repeat. The key element in medical practice is that “Collection is focused narrowly on information that will help to discriminate the relative probability of alternate hypotheses.”²⁹ So the best medical diagnosticians are gathering “just enough” data to form a hypothesis, then they test that hypothesis. New information can and will be discovered in the course of testing the hypothesis, which is then fed back in to the diagnostic model.

The great thing about a data driven approach is that we don’t have to first watch all the film of every player or gather all the available stats. We can start by noticing symptoms (hey, big college receivers seem to catch a lot of TDs), formulate a hypothesis (size and TD soring are related), test it (analyze size & scoring data for college and NFL players) and make a diagnosis (size matters). Others can re-test or challenge our diagnosis, and as new information emerges the diagnosis (prediction) can be refined. If we find more size and scoring related stats – maybe from next year’s batch of new players – we can run them through the model very easily to see if it’s still valid.

Riddle Me This

So our intrepid analyst has gone about gathering information in a manner consistent with focused hypothesis testing, now what?³⁰³¹ What type of new information should he pay attention to? Let’s distinguish four types of incoming data:

Noise:

• Additional detail about already accounted for variables. This type of data doesn’t increase the accuracy of the model. It just creates a false sense of confidence, which can be dangerous. For all of us, this is one of the central hazards of draft season: repeated exposure to the same information.

Maybe Noise, Maybe Signal

• Information about new variables not previously considered. This one takes some explaining. In a narrative driven mental model, new information often has little or no value, since the analyst is relying on a small set of observations³² to form their prediction. In an explicit data driven model, information about new variables may or may not be useful. While it’s technically true that all new information increases the accuracy, for example, of a linear regression, it’s also advisable to utilize parsimonious models³³. If a new variable doesn’t significantly enhance the accuracy of a model’s predictions, it may be ignored. Why? The more variables added to a model, the greater the number and complexity of assumptions within the model. This makes the model less generalizable and harder to test and adapt. For an example, revisit the medical diagnosis discussion: new data collection is only necessary if the research hypothesis was invalid.³⁴ At some point, more information no longer changes the diagnosis or the patient’s treatment plan. In other words, the “scientist must decide at what point the gain in the explanatory power of the model no longer warrants the additional complexity of the model.”³⁵ So when new variables come to light, they can be tested to see if they substantially improve the model. If yes, they can be incorporated; if not, they can be set aside. This is not the case in a narrative driven mental model that assigns value to every observation. Every observation must be accounted for, which either drives the mental model to such extreme levels of complexity and esotericism that it’s predictive utility is degraded.³⁶, or leads it to selectively pick out the puzzle pieces that form the pre-conceived picture for which the analyst was searching.³⁷

Signal:

• New information about observations already considered in the model. For example, maybe a player improved his 40 time significantly at his pro day vs the combine. Using the updated value in the model may materially affect its prediction.
• Information about the strength of relationships in your model’s variables. This is also important. The model (because it’s external and empirical) can be tested, challenged, and refined. Perhaps the initial model includes weight, speed, and draft position equally weighted. But additional research reveals that one variable should receive more weighting than the other two. This is what I meant when I said “the evolution of stats based analysis is always falling forward.”³⁸ This is also where a lot of data driven analysis can be improved; we may have a good idea of which variables are important, but not how much they matter.

Conclusion

So what have we learned? Traditional narrative-based approaches to analysis and prediction are fraught with drawbacks. The best approach to intelligence work – and player prediction – is to use an explicit, empirical framework to focus data collection relative to hypothesis testing. This process is easily replicated, better accepts critique and improvement, and yields more robust predictions.

————–

Find me on G+

Subscribe to the best.

1. emphasis added  (back)
2. Looking at you, Jack Ryan  (back)
3. Millen Alert  (back)
4. Kennerly & Mason  (back)
5. Epistemology  (back)
6. 1, 2, 3, 4  (back)
7. Kennerly & Mason  (back)
8. Money in the Banana Stand  (back)
9. CIA  (back)
10. Lookin at you, Jack Ryan  (back)
11. CIA  (back)
12. Penn State IRP  (back)
13. Police  (back)
14. NSA cell phone spying for $100, Alex  (back)
15. Dr Paul Slovic  (back)
16. By ‘confidence’ I mean their personal belief that their prediction would be accurate; not the ‘statistical confidence’ of the prediction  (back)
17. CIA  (back)
18. CIA  (back)
19. The feeling of confidence, not statistical confidence  (back)
20. Dr Paul Stover  (back)
21. Looking at you, Jack Ryan  (back)
22. RotoViz FTW!  (back)
23. Saying “paralysis by analysis” would be too cliché  (back)
24. Wikipedia bitches  (back)
25. Looking at you, Jack Ryan  (back)
26. Oops  (back)
27. and maybe not possible; roughly 25K college players, of which 3500 or so could be draft eligible  (back)
28. Elstein & Schwartz  (back)
29. CIA, emphasis added  (back)
30. Looking at you, Jack Ryan  (back)
31. Pool time!  (back)
32. And maybe doesn’t even realize which observations those are  (back)
33. Just Simple Enough  (back)
34. see what I did there?  (back)
35. Introductory Stats  (back)
36. What does wine tasting have to do with football?  (back)
37. To be fair, data driven analysts can also start with an end result in mind and find data to fit. The difference is that it’s much easier to refute bogus data-driven work than bogus narrative-driven work.  (back)
38. Interview with Zach Law  (back)