In researching recently for Veterans Committee ballots, I emailed official MLB historian and longtime baseball author John Thorn to see if he might know anything. Thorn replied, “Pete Palmer would know if anyone would” and put me in contact with him.
For more than 40 years, Palmer has been the person to go to for baseball questions even other researchers might not know. He is the researchers’ researcher, one of the wise old men of the Society for American Baseball Research, which he joined a month after its founding in 1971. Tom Tango, one of the co-authors of the influential 2007 work on sabermetrics, The Book, told me via Twitter, “Pete Palmer was our Yoda.”
At 77, Palmer remains active in baseball research and is father to three school-aged children, as he remarried after being widowed. Palmer and Thorn, who’ve written extensively together, just released a new edition of their classic 1984 book, The Hidden Game of Baseball, available via Amazon and elsewhere.
Truth be told, I’ve long wanted to talk to Palmer, and shortly after we began emailing about Hall of Fame voting, I asked if he’d be up for a phone interview. He was. Highlights of our hour and 40 minute conversation from Monday are as follows:
Baseball Past and Present: What’s it like being a dad at 77?
Palmer: I let [my wife] do a lot of the work, but I try to pay attention to them. I’m not some guy who’s sitting at the computer all day, typing away and ignoring everybody else. But they’re doing okay. My wife decided to home school the two boys this year, so she’s been pretty busy with that. Some days, they are a little reluctant to get into their studies, so that’s a bit of a problem. They’re, let’s see, third grade and six grade they’re in. We’re hoping maybe they can go back to regular school next year and give her a break.
BPP: Are your kids into baseball?
Palmer: Absolutely not. They’re into computers. They sit with their stupid little iPods, whatever you call them, the little handhelds playing games. It’s hard to tear them away from that sometimes.
I enjoy computer games as a break from doing the research. Sometimes it gets pretty tedious. Like, I’ve been working on Retrosheet. Retrosheet is going to have play-by-play of every game back to 1946 by this summer, and I’ve written a lot of programs to analyze the data and also look at the discrepancies. When you get back into the ’70s, there are a lot of mistakes in the official averages, and Retrosheet data is probably more accurate for the most part. But I try to look at everyone and try to figure out who’s right and what really happened.
Of course, when I started, we didn’t have all this stuff. But now, you can sit at your console and call up [lots of information.] One of the guys photocopied all the official averages which go back to 1905 in the American League and 1902 in the National League. You can look at newspaper accounts of the games for the box scores and stuff. It’s a lot easier to do than it would have been, say, even 10 or 20 years ago.
But it’s really kind of tedious. As I get older, I’m not quite as comfortable sitting down at the console for three or four hours and doing stuff. I used to to go to the Hall of Fame to get the records that weren’t in the guides, like the players of less than 10 games and stuff like that. I could sit there all day with my nose into the microfilm reader. But now, after a couple of hours, I have to kind of take a break. So I reward myself by playing a few games, Minesweeper or something.
The following came while Palmer and I discussed the work he’s doing now.
Palmer: Are you familiar with the player win averages which are developed by [Harlan] and Eldon Mills? They developed the probability of winning the game table based on the inning and out and the runners on base and the score at the time.
BPP: Yeah, I’m familiar with win probability.
Palmer: Yeah, right. So anyway, back in 1969, they got data from Elias which allowed [them to] calculate all that data. They did it again in 1970, and they didn’t find that much interest in it, so it fizzled.
We now have the capability of calculating those numbers back to 1946, so I’m working on that for a Research Journal piece which I’m going to finish at the end of the year which has the win probability differences for everybody 1946 to ’15, which is 70 years. One thing I did was I put in a park and decimal factor because obviously if you play in a good hitters park, your batting is going to be better. And if you’re in the National League, then the pitcher batting counts as part of the league average, so batters in the National League for a given level would have a better relative performance compared to the league average, so I had to factor that out. I’m not really sure how it’s gonna come out. Barry Bonds will probably be the leader, but I’ll have to see.
The one problem with player win averages is there are only a handful of really pivotal situations where the probability of winning the game can change by 20 or 30 percent on one at-bat. What happens is those particular at-bats get overvalued. I think I looked at Albert Pujols one year, and if you took his performance in his top ten stress-wise or game-changing position [at-bats], that like overshadows 200 of his lesser at-bats. Because lots of times, the game isn’t really on the line when you’re batting, but your performance is still important. Over the course of 10 or 15 years, that theoretically should balance out. But over the period of a season, you might get one key hit that would add like half a win to your total. The leading player for the season might be like five wins above average. So that’s a bit of a problem with it. Course, the player doesn’t really have a choice of when he gets up. He just tries to do the best he can every time.
Still, it’s interesting, and there’s a pretty high correlation between that and your normal ratings. In fact, Dick Cramer did a study years ago on the Mills Data where he looked at players in 1969 and ’70 and compared their player win average to their nominal batting stats. Theoretically, if you were a clutch hitter, then your player win average should be higher than your normal situation where every bat counts the same. And he found no correlation whatsoever, which is what I expected, because I’m [in] the ‘Clutch hitters don’t exist’ camp.
Cramer and I did this study a couple years ago where we had everybody back to 1950, I think, 1950 to 2005, something like that, and there was no significant difference between the player’s overall record and his record weighed by the probability of winning the game. And that hasn’t really changed. So that’s one of the things I’m looking at when I do this. But I don’t know, are you a clutch hitter advocate or not?
BPP: Um, I mean honestly, I feel that in most things in life, not even just baseball, I feel like there’s people who perform better under pressure and have different motivations. But I also understand one of the arguments against clutch performance in baseball, which is the idea that if a hitter could do that, why aren’t they doing that every single time?
Palmer: Right, that was one of Cramer’s points. Because, if you look at it, the guy’s up there about 10 minutes a day, at the plate. I know there’s stressors and you’re tired and stuff, but it shouldn’t really be that hard to concentrate for 10 minutes a day on your job… I would think that if there was clutch play, [it would] more have to be the pitcher because the pitcher actually is working hard. In the old days when everybody used to pitch a complete game, they sort of had to pace themselves a little. So you’d think, ‘Well, two outs, nobody on, I’ll just try to get the ball over and see what happens.’ I didn’t find any evidence of clutch pitching either. But you’re right, why don’t they try all the time?
BPP: What do you make of defensive independent pitching stats? Do you think there’s anything to it?
Palmer: I didn’t have a lot of faith in that. I think that if a pitcher’s a good pitcher, you’re not going to get as many hard-hit balls off him, and if it’s not a home run, the hard hit balls are going to be in the field. We have hit locations and hit types back to 1988, which is like 25 years, more than 25 years. It says ‘line drive,’ ‘ground ball,’ ‘fly ball,’ ‘hard hit,’ ‘medium hit,’ ‘soft hits’ and the locations. If you hit a fly ball and it’s not a home run, the odds [are] it’s going to be an out. Of course, a pop up is almost a clear out. But line drives, they got a pretty good chance of dropping in.
I think what [Voros] McCracken’s problem was that he didn’t realize the variations due to chance on hits. If you hit a bunch of line drives, a certain percentage are going to be caught, but there’s a fair amount of variability in that. If you look at individual pitches, you see, well, the differences of outs on balls in play is not varied as much as maybe you would think without actually looking at how much it should be.
But the problem with DIPS is, if you prevent a home run, then your outs on balls in play go down because now, the balls rattling off the outfield wall somewhere. Whereas if it goes into the stands, it doesn’t count. If you turn a double into a single or a triple into a single because of your superior pitching, that doesn’t affect your outs on balls in play at all. And if you strike out somebody– strikeouts are a fairly important part of being a good pitcher– your outs on balls in play go down again because the out doesn’t count. So the only time that a good pitcher really makes a difference on outs on balls in play is when he turns a single into an out. When you don’t look at walks and strikeouts and home runs, you’re actually minimizing a difference between a good pitcher and a bad pitcher. And therefore, the gap in that category is going to be artificially low because some of the factors that would make it higher are not counted…
[McCracken] said there wasn’t a great amount of correlation from season to season. But as I said, the variations due to chance and everything in sports, baseball in particular, is a lot higher than people think. Your average could drop 60 points from one year to the next, and it’s not really statistically significant because 500 at-bats isn’t that many at-bats to verify what your current batting average should be. I think the average difference in batting average between seasons is something over 30 points, so 60 points is really just on the borderline of being statistically significant.
Even team wins. The variation due to chance from one season to the next on team wins is about nine. So what that says is, you can drop 18 wins in one season, and there’s still a 5 percent chance it was just luck.
BPP: Yeah, I feel like teams can swing from 70 to 90 wins pretty easily.
Palmer: I was amazed, the Red Sox were picked by Sports Illustrated to be the eastern division champs this year. That seems strange to me. I’m a Red Sox fan, you know, I go back to the ’40s, but I don’t see them being eastern division champs.
I never looked it up, but their swing over the last three years has got to be one of the biggest swings, from down to up and then down again. I don’t know if any team’s ever done that much. I never got around to looking it up, but I’d be surprised. They swung about 25 games, I think, plus or minus. That’s pretty unusual.
BPP: Bobby Valentine was a pretty terrible manager.
Palmer: He got off to a wrong footing and never really recovered. I think managers are overrated in baseball anyway. It’s probably easier to do a crummy job. There’s more downside to being a manager than upside, I think.
BPP: I don’t feel like a manager has much effect, but it seems like a really bad manager can still do some damage.
Palmer: Yeah, yeah, that was my point. They all do about the same thing. The nuances and strategy, if you look at it mathematically, the so-called optimal strategy might be a little better than what most people, but the differences are so small, you’d be hard-pressed to get an extra win out of intentional walks and stolen bases and sacrifices and all that stuff over the course of a season by altering your strategy, I think. Or even the batting order. The difference between an optimal batting order and a normal batting order is probably less than one win per season.
All that stuff isn’t important, but I think it is important to try to keep everybody working toward the same goal. I think that’s probably the biggest job for a baseball manager. Whereas in football, the coach I think it makes a huge difference.
Palmer has done statistics work for the New England Patriots for more than 40 years– he actually prefers watching football to baseball– so our conversation veered away from the national pastime for a bit.
BPP: Is it ever frustrating to you that football has been so conservative and slow to embrace newer kind of statistical thinking?
Palmer: Yeah, yeah. Well, the trouble is if you do the conventional thing and it doesn’t work, nobody can criticize you. But if you do something weird, and it doesn’t work, then it looks bad.
The Patriots had a game a couple years ago in the playoffs where they had a 4th and 5 or something like that, and they faked the punt and tried to go for it and then they didn’t make it. It might have even been against Denver, I can’t remember. It was a good play. It didn’t work, but you don’t even know what the success rate should be because they never try it.
Some coach in high school, I remember reading an article in Sports Illustrated about it, a few years ago, he decided punting was stupid. So he vowed never to punt again. It doesn’t matter where they are, on fourth down, they go for it. And he’s had some luck with it, though that seems a little extreme. Even 4th and 5 inside the 10-yard line, mathematically it’s better to go for it.
BPP: Yeah, you’re giving yourself 33 percent more downs if you just always go for it.
Palmer: See one thing that happens is, if you kick a field goal, then the other team’s going to get the ball back around the 20 or the 25 after the kickoff, whereas if you go for it and don’t make, then they’re going to be down on the one-yard line. So that’s like 24 yards, and on average, 24 yards is worth about two points. Theoretically, if you kick a field goal, you’re giving away two of the three points you just got by field position after the kickoff. So it’s really only worth one point. Whereas with a touchdown, then it’s seven points and then again you’re giving away two points, but that’s the difference of five points to one point instead of seven points and three points, which is a difference.
There’s a guy, Aaron Schatz who has FootballOutsiders.com… Aaron and I have talked about that. He says, ‘They’re just not going to do it. It doesn’t matter. You’re never going to talk them into it.’
BPP: Have you tried to affect any change working in the NFL?
Palmer: No. My friend Bob Carroll, who founded the Pro Football Research Association, and I wrote a book called The Hidden Game of Football about 25 years ago, which had a lot of that stuff in there. It didn’t really have any impact on anything. They’ve been keeping track of fourth down conversions for 30 years now. It really hasn’t changed. It’s about one per game, per team, and they make it about 40 or 45 percent of the time. There’s been no movement of that in 30 years.
BPP: Why do you think it’s changed in baseball and not football?
Palmer: Baseball hasn’t changed that much. Obviously, rating on-base average more importantly has been a big change. But the strategy changes really haven’t changed that much. Some teams bunt less, and Earl Weaver used to hate intentional walks, but for the most part, there’s not a lot of difference.
One thing in stolen bases is, it’s become more specialized. The guys who are good at it steal a lot, and the guys who aren’t hardly ever steal. That’s been sort of gradually coming into the game. Everybody adopted the Tony La Russa relief pitching strategy around 1990. Nobody has really deviated from it. So that was one change but that wasn’t based on sabermetrics. That was based on La Russa being somewhat successful with it.
One of the big changes, just this year was people are using a lot more defensive shifts. It used to be just a couple guys that would get shifted down, Ortiz, a few others. But now, they’re doing it a lot. For the most part, I think it’s based on the work that John Dewan has done with Baseball Info Solutions where they plotted hit patterns for everybody in the league.
My friend Steve Mann and I had a contact with the Phillies back in the ’80s where he was somewhat familiar with the Phillies’ front office so he got us in there. We were plotting pitch locations and hit locations. They had a guy there, Jay McLaughlin who’s still there, who wrote a lot of programs to plot that stuff up. I don’t know really how much they used it, but that data’s been around for like 25 years. They only really started using it recently.
BPP: Do you think the amount of competitive parity these days is a sign that baseball’s gotten better?
Palmer: You sort of have to think that. If you go back to Babe Ruth’s day, the number of North American white players in the major leagues are about the same today as it was then, but the population of the United States, of course, is much greater. I think the Dominican Republic ranks third among [countries] as far as number of players in the major leagues, something like that. The Japanese players are coming over, of course. There’s a much bigger pool of talent.
There are more sports, and you can make a lot of money. Back in Ruth’s day, you couldn’t really make that much money in any other sport, maybe prizefighting or something. Now you can make a lot of money playing basketball, hockey, or football. But there’s never really been a great player in more than one sport, except for I would say Bo Jackson. So I don’t think there are that many football or basketball players who could have been exceptional baseball players. They might have been average baseball players, but I don’t think the other sports are really taking people away from baseball.
Other interviews with people from the baseball research community: Robert Creamer, Rob Neyer, Joe Posnanski, Dan Szymborski, John Thorn.