In-Depth: Does Spring Training team performance mean anything?

Every March a similar debate arises between two distinctive camps, those who look at the Pittsburgh Pirates’ performance at Bradenton as a predictor of the season to come, and those who claim that there is no inference worth taking from Spring Training play.

Each side has valid points. The Pittsburgh Pirates are trotting out, at least for the first few innings, their starting talent.  The common criticism heard in many circles is this: If the club isnot winning games against other team’s major league talent during Spring Training, how can they win games in the regular season?

On the other hand, managers and coaches don’t necessarily want to win every game of Spring Training, preferring instead to see how particular players perform in certain scenarios, testing a player’s ability to do one thing or another and so on. The results are not necessarily representative of how the team will play once they make their way back north.

Fortunately, this is a quantifiable proposition. If a Major League team’s Spring Training records are predictive of what their record is during the following regular season, we can test for it and then make a prediction based on that Spring Training record, of what their regular season record will be.

To test this, I pulled each team’s pre-season and regular season records from 2013 to 2017 then regressed each team’s regular season wins as a function of their spring training winning percentage. This allows us to test for a statistically significant predictive ability of a team’s ST record on their regular season record. We can graph this out with something like this:

The takeaway here is that there is a statistically significant correlation (p = 0.027, R²=0.032), between Spring Training winning percentage and the following regular season wins. The problem is that it is a pretty weak predictor. It basically means that using a team’s ST record to “explain” what their regular season record will be, is only 3.2 percent “better” than just guessing every team will finish up 81-81, which makes it not all that helpful.

Anecdotally, the 2016 Cubs, who won 103 games, had a Spring Training winning percentage of .367 (that’s a 60 win clip during the regular season), in fact, only three of the five 100+ win teams over the previous five seasons had ST records over .500.

Perhaps a clearer way of putting this is that if you use a team’s ST record to predict their regular season record, you’ll probably be right more than you’re wrong, but you’ll still be wrong a lot too. Ideally, we’d like a predictive analytic with a smaller p-value and higher R², as that would make it a better predictor.

After poking around through the data for a while, I decided to look at the relationship between a team’s ST run differential (Runs Scored minus Runs Against) and their regular season wins.

This was better than Spring Training record alone (p=0.004, R²=0.055), but still not a lot better than just guessing .500. To put this into perspective a team’s regular season run differential correlates to regular season wins with a p-value that is very close to 0 and R²=0.852.

I put these findings out on twitter and received an interesting idea in return:

https://twitter.com/jfn514/status/974742125928697857

A better indicator?

Since the Pittsburgh Pirates and most other teams generally pull their starters after the first five innings or so during Spring Training, finding their run differential after this time would be more likely to predict what their starting players will do during the season, and eliminate the statistical noise in the data created by double and triple A players playing about half the game. After a good bit of pulling and cleaning up data on Spring Training games over the past 5 years, I was able to establish each team’s run differential for their first 5 innings (referred to as “RD5”) for each Spring Training season. Comparing that 5 inning RD to their ensuing regular season wins gives this:

In some sense, this is just a more granular look at what was presented above, but it’s effectively a cleaner version. Since we eliminate the “noise” created by depth options/minor leaguers playing, this is a much better look at the Major League talent; this is reflected in the correlation (p= 4.92e-5, R²=0.105).

If we use the Pittsburgh Pirates’ RD5 at the time of writing this, -27 runs, it predicts them as a 77 win club. If we assume the Pirates will continue to play at this same run differential pace over their remaining 9 games, they would have a -38 RD5, projecting them as a 75 win team. In either case, that would put the Pirates as a few wins worse than the Reds for the basement of the NL Central.

The trouble is that this isn’t necessarily predictive of what the Pirates have done in the past 5 years. In 2013, for instance, the Pittsburgh Pirates had the worst RD5 in Spring Training, clocking in at -47 runs, nearly 10 runs worse than their current pace; that team ended up winning 94 regular season games and making the playoffs. Similarly, in 2015 the Pirates had a Spring Training -19 RD5, which would have projected them as a 78 win club, instead they went on to win 98. On the flip side the Pirates best pre-season RD5 over this 5 year span was in 2017 with +29, projecting them at 85 wins, they went on to underperform that projection by 10 games.

Perhaps the only thing that matters is a team approach

[perfectpullquote align=”right” bordertop=”false” cite=”” link=”” color=”” class=”” size=””]On the other hand, managers and coaches don’t necessarily want to win every game of Spring Training[/perfectpullquote]

This could just be the nature of predictions; random variation, by definition, is a very difficult thing to predict. The errors are simply the effect of randomness. On the other hand, the Pittsburgh Pirates may take a different approach during Spring Training than they do in the regular season. The club may treat Spring Training as just a glorified live-ball practice. If a player needs to work on a particular aspect of their game, like a particular pitch, a type of defensive shift or hitting against a type of pitcher, they do it regardless of its impact of on the team’s chances of winning that game.

The Pittsburgh Pirates don’t seem to care about team performance as much as individual performance during the spring. As a few examples; Tyler Glasnow and Steven Brault are getting regular starts despite being destined for the bullpen this season. Sean Rodriguez and Adam Frazier are slotted to be the backups, but are still getting pretty regular time in the outfield to get comfortable. David Freese is getting pretty significant time at first base, to help him be ready to play a position he only logged 15 innings at last season and so on.

The purpose of playing these players like this isn’t to win Spring Training games; it’s to give the team a better shot at winning regular season games. If Glasnow and Brault can provide high quality spot starts and eat innings out of the pen; if Rodriguez and Frazier are comfortable and productive playing and giving the regular outfielders rest; if Freese can be a plus defender and bench bat, the Pirates can win more games than they otherwise could without this production from depth players.

To answer the question posed in the beginning of this article: can we infer anything about a team’s regular season from their spring training? The answer is not really. While using a team’s run differential through the first 5 innings is better than guessing blindly, it still isn’t definitive by any stretch.

What’s more important is looking at how the players are playing; if guys like Glasnow, Freese and players like them are getting quality innings in Spring Training, then the Pirates are a deeper team for it and have better chances of winning ballgames come the summer. What exactly “quality time” is, is more of a qualitative judgment; some players have great springs and can’t translate that to the regular season, others don’t get up to speed until May or June when they take off. Putting together a roster for the beginning of April is much more of an art than a science; similarly assessing that team coming out of ST is more of an art than a science. If you’re an optimist or a pessimist about the Pittsburgh Pirates’ chances in 2018, then you can find things to be optimistic or pessimistic about in the team’s Spring Training performance, but the impact of that performance on the regular season, aside from getting player ready to play, is actually minimal.