What Exactly is a “Quality Stat”?

Sometime during last season, I discovered the Cold Hard Football Facts website. Their stats look promising, and I’ll probably be analyzing their predictability in the next couple months. However, about a month ago, as their Twitter feed kept touting their latest “Quality Stat”, Passer Rating Differential, it hit me: I (at least slightly, more on that below) disagreed with their notion of what a quality stat actually is.

In their list of top Passer Rating Differentials, they define a quality stat as “[having] a direct correlation to victory”. That’s a good start, and to test this, I took Passer Rating Differential from the past three years and correlated it with winning percentage.

Lo and behold, it does indeed correlate pretty well with winning. The r-squared value of .638 (1 would be perfect correlation) is not bad at all. You know what correlates even better with winning? Point Differential.

The r-squared value of the correlation between point differential and winning percentage is 0.835, which is even closer to 1. I have our latest Quality Stat: Point Differential!

In their June article touting Passer Rating Differential, they have an informative chart listing the top 25 teams in Passer Rating Differential, and noting that 14 won titles and three others lost in a championship game. Well, you can do the same thing with Point Differential:

Out of the top 20 teams in Point Differential, nine won a championship, and six others made it to the title game and lost. That’s most definitely a “Quality Stat”, at least according to CHFF’s definition.

In my opinion, a quality stat should correlate with winning, but it should also provide some type of new insight, i.e. be able to tell you why certain teams are winning. Point differential, of course, doesn’t provide any insight at all, because it tells you that if you have a good offense and a good defense, you will win games. Passer Rating Differential tells you that have a good passing offense and a good passing defense, you will win games. It might help dispel the notion that a good rushing attack is most important, but that’s about it.

The point of stats such as those from Football Outsiders and Advanced NFL Stats is that not all yards are created equally, and the yard from the 1-yard line to the goal line is not actually worth six times more than the other yards (stupid fantasy football). That’s all you really need to know to understand their stuff, and I dare say that the applications for their stats are more intuitive than CHFF’s “Quality Stats”. With DVOA or EPA, you can break it down by play type, and make conclusions like the Steelers are 30% better than an average team on third down and long, or running plays on first down added 7.5 expected points. You can also give a team’s passer rating on third down, but it’s useless unless you know other teams’ passer ratings on third down (of course, you can say the same thing about expected points).

Having bashed the quality stat concept for several paragraphs, I will say that CHFF’s Quality Stats power rankings are very informative, since they combine several measures of team ability and provide the insight that you have to do several things well in order to win, and I definitely recommend looking at those throughout the year.

The holy grail of football analysis is a stat or set of stats that will tell you exactly how valuable a player is to his team, regardless of the team around him. Baseball has that pretty much down to a science at this point, but football has a long way to go. The intermediate step, which several websites are close to achieving, if not there already, is to have a stat or set of stats that explain why teams win. I will be watching these sites in the future and hope to someday have an inspiration that allows me to help develop a statistical system that truly values individual players independent of the offensive or defensive scheme they play in. That will be a quality stat indeed.