People are often curious about the predictive ability of Sim Scores and sometimes the counter-intuitive projections that the Sim Scores come up with seem like they can’t be right. To address the first issue, I’ll say that I’ve run backtests of Sim Score predictions and pitted them against a linear regression and the Sim Scores have equaled or beat the linear regression, depending on position group and sample size. As to the second point, the thing that the Sim Scores pick up that sort of hurts our brains is that there are in some cases non-linear relationships between variables, and variable interactions, which the Sim Scores can capture, but that seem counter-intuitive. For instance, Sim Scores like young, big, pass catching running backs a lot. Well lots of people like those running backs, but the Sim Scores like them probably more than anybody else. But that’s because lots of projection systems are going to have a tough time accounting for things like RB age, or the interaction that RB age might have with weight. Sim Scores are lazy in that they just say “if these guys are relatively similar, then using 20 similar guys to forecast another similar guy should yield a forecast that will be good.” Doing that allows us to bypass time we might have spent figuring out an upside down U-shaped age curve, or even trying to figure out the interaction that a bunch of variables might have among each other. It’s true that Sim Scores will have a difficult time dealing with players that might exist at the edge of the distribution (like Adrian Peterson) but it’s also the case that using another projection system trained on data that included no similar observations is also barely better than guessing.
But the easiest way for you to understand what the Sim Scores are saying is probably to play around with them to see how they change when you change various variables. In order to do that I’ve created a “lab” app that will let you create a Theoretical RB. You can change the variables and see how the forecast changes in Year N+1. I call it a lab because you can experiment and learn for yourself how these things work. The controls will just change the values that the similarity search runs off of. Hope this helps you learn how the Sim Scores work.