A new method of evaluating offensive line play

On Tuesday, STATS, Inc. released a new stat called the “New York Life Protection Index” (take a guess which company is sponsoring the stat). It claims to measure the offensive line’s ability to protect the quarterback, and looking at the current standings, it certainly passes the eye test, with the Steelers at 29th out of the 32 teams. Other places, such as Ed Bouchette over at the Post-Gazette and Dave at Steelers Depot, have already given their takes on this, but I thought I’d add my two cents after the jump

The index is calculated using a “proprietary formula”, meaning no one except the people at STATS know exactly what it is, but they list the components of the formula:

• Length of a team’s pass attempts (presumably before the catch)

• Penalties

• Sacks allowed

• Quarterback hurries and knockdowns

Based on the name, the stat is supposed to measure QB protection, yet those first two components seem to be much less important than the other two in actually doing that. I decided to compare the rankings in the index with Football Outsiders’ Adjusted Sack Rate to see if there were any large differences, and there were:

A common thread among the teams whose NYLPI ranking is higher is that they (except for Seattle) all have high-powered offenses in the top half of the league in both FO’s and Advanced NFL Stats’ passing rankings. Also, all the teams whose NYLPI ranking is lower (except for KC) are in the bottom half of penalty rate, as measured by ANS. Pretty much expected, given the formula, so you know that their formula is what it says it is.

Dave’s main criticism of the stat is that it doesn’t consider all the factors, especially how long the QB holds onto the ball (something we all know well with Ben Roethlisberger). I tend to agree with this; even the Football Outsiders site admits that their stats don’t completely separate out what the OL is doing from what the QB is doing.

Dave looks at the stat in comparison to his own film-based way of evaluating the o-line, and concludes that the stat is garbage. I wouldn’t call it garbage, though I don’t think this index would be released to the public if it wasn’t sponsored by someone. The main difference between statistical approaches and game film approaches is results versus process. All the stats out there, even the advanced ones, look at results, while if you want to see whether the process was done correctly, you have to watch the game film. Even Football Outsiders has “game charters” watch every single game and look for things that aren’t listed in the official play-by-play. In the end, it’s a matter of how you want to measure things, and it’s something that will be interesting to look at in the future.