Tuesday, December 11, 2007

Zimbalist reviews Bradbury, Bradbury responds

Famed sports economist Andrew Zimbalist has posted a review of two recent baseball economics books: Vince Gennaro's "Diamond Dollars," and J. C. Bradbury's "The Baseball Economist." The Bradbury review is of more interest to me, since I haven't finished Gennaro's book. Plus, Bradbury has responded to Zimbalist on his website.

Here's
Zimbalist's review; here's Bradbury's response.

It seems to me that Zimbalist makes some good points, but also some very questionable ones. Bradbury defends himself well.

And my sympathy is with Bradbury, because Zimbalist doesn't know much about sabermetrics, and, for that reason, some of his criticisms are badly miscast. On various points, it's clear that Zimbalist is oblivious to most of the sabermetric progress of the past thirty years or so.

Take clutch hitting, for example, the subject that has arguably been debated the most of any controversy in the field. It seems like Zimbalist has seen none of that work. When Bradbury argues that clutch hitting talent is an illusion, Zimbalist responds

"There you have it – there is no such thing as clutch hitting. This is an awfully linear, materialist view of the world where a player’s emotions and his state of physical depletion over a 162-game season play no role."
This kind of intuitivist, naive response is what you expect from sportswriters, not from people who study sports economics for a living. But Zimbalist seems unaware of any of the body of literature on this question.

Here's another one. Bradbury talks about OPS, and how a better version of the formula would give more weight to OBP, perhaps by a factor of 3 (as Michael Lewis quotes Paul DePodesta in Moneyball). Zimbalist writes,

"... SLG ... is a much higher number than OBP. The coefficient, therefore, will necessarily be smaller on SLG. If elasticity is used instead of the estimated coefficient, OBP is 1.8 times greater than SLG."

Here, Zimbalist is trying to criticize Bradbury for the way he casts the question, arguing that the coefficient is an inappropriate measurement here. But if he were familiar with Moneyball, or the debate on OPS, he'd have known that the question DOES refer to the coefficients, and, yes, we are indeed aware that SLG is a higher number. Again, it's apparent from his comments that he has no idea there's already an extensive literature on the subject.

And here's one more:

"[Bradbury argues] that a pitcher’s ERA from one year to the next is highly variable, but that a pitcher’s walks, strikes and home runs allowed are more stable over time. The inference is that ERA depends more on outside factors, such as a team’s fielding prowess, and, hence, is a poor measure of the inherent skills of a pitcher. While there is something compelling to this logic, it seems caution is in order. First, a pitcher’s skills may actually vary from year to year, along with his ERA, as other factors change, such as, his ballpark, his pitching coach, his bullpen, his team’s offense, the angle of his arm slot, his confidence level, etc. This variability does not mean that the skill is spurious. Second, if all we consider is strikeouts, walks and home runs, what are we saying about sinkerball pitchers who induce groundballs or pitchers who throw fastballs with movement or offspeed pitches that induce weak swings and popups?"

That last sentence, about pitchers inducing weaker balls in play ... well, what we are saying about it is the DIPS theory. And that chapter of Bradbury's book does include an extensive discussion on DIPS ... if Zimbalist did indeed read it, you can't tell by his argument here. Bradbury rips into Zimbalist for this, with a lot more restraint than I would. Also – and this Bradbury does not mention – is that it's not just "outside factors such as a team's fielding prowess" that makes ERA unreliable. It's mostly just luck – whether the hits, walks, and home runs are bunched together or not. I'm sure Bradbury, or any one of countless bloggers and writers in the field, could have told Zimbalist this.

Anyway, as I said, Bradbury defends himself against Zimbalist quite well. For my part, though, I have to say that it's disappointing that Zimbalist, who is so respected in the realm of sports economics, would know so little about sabermetrics. After all, sabermetrics is an established scientific discipline, and one quite substantially impacts his own. Moreover, Zimbalist seems unaware that he is unaware. You'd expect a reviewer to be well-versed in the subject he's reviewing, but that doesn't seem to be the case here.

Zimbalist's review has been published in the "Journal of Economic Literature," an academic publication. This is unfortunate. I don't know much about how things work in academia, but it does seem that Bradbury's reputation will take a unfair hit -- at least on these sabermetric points, on which Zimbalist's less-than-fully-informed criticisms are way off the mark.




Labels: , , ,

18 Comments:

At Tuesday, December 11, 2007 4:50:00 PM, Anonymous Anonymous said...

I agree that Zimbalist's sabermetric review was poor. There was definite missteps in JC's sabermetric work, which I tackled here:
http://www.insidethebook.com/ee/index.php/site/comments/the_baseball_economist/

Zimbalist was simply out of his element on this topic.

I also think that Zimbalist ignored the basic focus of JC's book, in that it's about economics, with illustrations from baseball, rather than explaining baseball through economics.

That said, when Zimbalist talks, I listen.

 
At Tuesday, December 11, 2007 10:17:00 PM, Blogger Phil Birnbaum said...

Tango, your link is incorrect ...

 
At Tuesday, December 11, 2007 11:42:00 PM, Anonymous Anonymous said...

Phil:
I agree that many of Zimbalist's criticisms appear to be off the mark (though I haven't read JCB's book). And the review actually reads like Zimbalist was out to get these guys, rather than do an even-handed review (don't know if that's true). But I'm puzzled by your highlighting of their disagreement on the relative value of OBP and SLG. I think Tango (and others) have shown quite convincingly that one point of OBP is worth about 1.8x -- not 3x -- one point of SLG. Do you disagree?

 
At Wednesday, December 12, 2007 1:14:00 AM, Blogger Phil Birnbaum said...

Hi, Guy,

No, I don't disagree at all. If Zimbalist had said that 1.8 was the right answer to the coefficient question, he'd have been right. But what he said was that because SLG is bigger than OBP, the coefficient is not the right way to answer the question.

And that's wrong, because the question was ABOUT the coefficient.

I may have misinterpreted what Zimbalist said in that quote, but it reads to me like he was talking about something other than what the discussion is about -- which is what a point of OBP is worth relative to a point of SLG.

 
At Wednesday, December 12, 2007 1:21:00 AM, Blogger Phil Birnbaum said...

Of course, I may be wrong. Zimbalist writes, "If elasticity is used instead of the estimated coefficient, OBP is 1.8 times greater than SLG." I don't know what "elasticity" means in this context, but maybe he's actually fixing the flaw in Bradbury's work. Can someone clarify?

[cross-posted to BTF]

 
At Wednesday, December 12, 2007 3:58:00 AM, Anonymous Anonymous said...

I believe that what Zimbalist said about elasticity is coherent.

I think Bradbury was very wrong to accuse Zimbalist of borderline dishonesty. That is a weaselly way to say that Zimbalist deliberately misrepresented his work without quite committing himself to the slur. The lone example that Bradbury offers is regarding public subsidies for stadiums for rival leagues. Now it is not clear whether this was Bradbury's own formulation of the barrier to entry earlier in his chapter, or Zimbalist's, but Zimbalist was clearly talking about stadium construction subsidies as a barrier to entry and Bradbury in his reply talks about USFL stadiums using [sc. already existing!] public stadiums. [No new stadiums were built for the USFL, if the team histories in wikipedia are reliable.] The text from TBE which Bradbury quotes in rebuttal does not convict Zimbalist of misrepresentation at all, because Zimbalist's point is about stadium construction.

In general, while Zimbalist's review is harsh, apart from his dubious remarks on clutch hitting, it looks to me like he generally has the better of the argument. I certainly do not agree at all that Bradbury has defended himself well. When you read Zimbalist's criticisms and carefully compare Bradbury's replies, they usually do not answer the actual criticism.

Here are some more examples:
1) Rent seeking.
Zimbalist mostly describes B's position, differing twice a) "Bradbury suggests that sometimes the managers succeed in this endeavor, though the statistical evidence he presents is too weak to support his claim" b) "the fans, according to Bradbury, have their utility lowered because they have to spend a few extra minutes at the game due to these fits of managerial distemper. Well maybe, but it is also possible that the fans enjoy managerial protests." B hides the first criticism by elipses, responding as if only to the second criticism by claiming that "The point of the chapter was to teach the concept of rent-seeking using baseball instead of typical boring classroom examples." But giving a statistical argument in favor of managers' success in influencing umpires goes beyond teaching the concept of rent-seeking. Here B does not fairly portray or rebut Z's remarks.
2) Advantage of big city vs small city.
Z cites what he says is B's conclusion "the advantage [sc. of big cities] appears to be slight and virtually meaningless", but calls his argument sloppy and notes that B's "simple regression finds that variance in city size accounts for 40 percent of the variance in win percentage over a period of years." He then mentions 3 factors B should have considered, including a team owning its own RSN. Z also claims B "misapprehends the functioning of the amateur draft and overlooks the unequalizing effect of the posting system with Japanese baseball." B responds by focussing on the word "simple"; he defends himself by listing several variables he did consider but left out. None were among the 3 mentioned by Z. B then addresses just one of them. "I admit to not including a dummy variable or interactive term for whether or not a team owned its own RSN. This is endogenous—having an RSN to generate money that leads to wins is something that any team could do, regardless of its size." Calling ownership of an RSN endogenous doesn't make it so. Perhaps I am putting too much faith in wikipedia, but the article there on regional sports networks does not seem to support this view. It appears from that article that only 5 baseball teams currently have an ownership share in an RSN, (2 very recently, postdating the seasons used in B's analysis) and 2 more teams have owned RSNs which failed. Later on in his rebuttal, B actually claims "...RSNs... most teams now have them."

3) scouting and scouting methods
I'll paraphrase here: Zimbalist objected to Bradbury's suggestion that "old scouting methods" may disappear and be replaced by sabermetric approaches. Zimbalist says that no team is considering replacing any old scouting method; sabermetric approaches are and will be supplemental only. Bradbury in rebuttal quotes a passage 3 pages earlier in the book in which he denies that traditional scouting will be replaced, but does suggest that the scouts' role may be modified and reduced. This is needless heat over the word "may". Zimbalist believes that current scouting methods will all be retained unmodified and Bradbury does not. Bradbury's denial that scouting itself will disappear is not germane to the narrower point of whether some scouting methods or scouting functions will be modified and reduced. Again , Bradbury says that Zimbalist has misread him, but the truth is that Bradbury has misread Zimbalist.

Also, Bradbury snipes at Zimbalist thusly:
"PROD is the sum of OBP and SLG—yes, that is OPS—yet Thorn and Palmer (1984) are not credited."
I pulled out my copies of Baseball and Billions and Total Baseball(2nd edition); there is a grain of truth to Bradbury's complaint. Zimbalist in his chapter, appendix A and footnotes did not explicitly indicate that PROD was defined and its values taken from Total Baseball [Thorn and Palmer 1991], but the 1989 edition of the book is mentioned in the appropriate footnote for a slightly different point.

And I think Bradbury is right in his rebuttal to Zimbalist's charge of inconsistency about the difference between clutch hitting and clutch pitching.

I don't care for Bradbury's petulant style of response to criticism. We've seen this before.

 
At Wednesday, December 12, 2007 9:33:00 AM, Blogger Phil Birnbaum said...

Hi, Joe,

What Zimbalist said about elasticity may be coherent, but it is irrelevant to the argument Bradbury was making.

I didn't mean to imply that Bradbury got the better of all (or even most of) the arguments, or that Zimbalist is out of line everywhere. Perhaps "defends himself well" was the wrong choice of words; "defends himself vigorously" was more what I meant.

What I *do* still mean to say is that on the specifically *sabermetric* criticisms, Zimbalist is way off line because he doesn't understand what the debate is about. And it doesn't matter what the elasticity of OBP is, the fact is that Bradbury is talking about coefficients, that we've been arguing about the *coefficients* for years now.

 
At Wednesday, December 12, 2007 9:59:00 AM, Anonymous Anonymous said...

Phil, unless you corrected it, link is fine.

For the sabermetric stuff specifially, Zimbalist is definitely out of his element. JC however, while he's ok, is not very qualified (as I detailed in my link). JC could have used a peer review, and so could have Zimbalist.

Both, however, seem very unpliable. For professors, they make lousy students.

A person can be confrontational or aggressive as I may appear to be, and others too. But, we are not Bush/Rove-like in our stance.

 
At Wednesday, December 12, 2007 10:22:00 AM, Blogger Phil Birnbaum said...

Sorry, link is indeed fine. I must have screwed up the first time.

I didn't mean to imply that JC's work was not deserving of criticism -- and you know I didn't mean to say that, you've seen my book review. I would agree with you that both JC's book and Zimbalist's review could have used some more peer review.

However: the issues with JC's work are that he got some things wrong. On the other hand, the issues with the sabermetric portion of Zimbalist's review is that he's not familiar with the field of study, and criticizes anyway! This would not be an issue if he was right. Even if he were wrong, which he was, it would be less of an issue if the criticism weren't so harsh, and it would be even less of an issue if Zimbalist had acknowledged that he may be wrong on these things because he doesn't do sabermetrics.

A reviewer, and the publication who prints him, has a very strong obligation to the author being criticized. This is especially true when the critic is renowned as one of the top researcher in his field, and the reviewee is not. This review is an expert reviewing a non-expert. Zimbalist's review carries a lot of weight.

I can't speak for the economics portion of the review (see Joe Arthur's comment for that), but, on the sabermetrics, Zimbalist simply did not know what he was talking about. On these points, he was just plain wrong, in a very obvious way -- obvious, at least, to anyone who follows sabermetrics. And that is very serious, and independent of how good JC's book is in general.

 
At Wednesday, December 12, 2007 11:28:00 AM, Anonymous Anonymous said...

I agree with the thrust of your argument. Zimbalist is an expert in one area to which he offers his opinion, and also offers a strong opinion on an area that he knows little about.

 
At Wednesday, December 12, 2007 12:10:00 PM, Anonymous Anonymous said...

Phil:
It's worth noting that despite his quibbles re: OBP/SLG and clutch hitting, Zimbalist actually says explicitly that it's the sabermetric chapters of the book that are the strongest. For JCB's sake, I certainly hope that isn't the case, as his ventures into sabermetric analysis have been almost uniformly unimpressive. I think the real lesson here is that sports economists -- including both Bradbury and Zimbalist -- still have a lot to learn from the "amateur" sabermetricians. (And this can even impact the quality of the more traditional economic analyses, as in your example of Bradbury failing to understand the importance of replacement level in valuing players.).

 
At Wednesday, December 12, 2007 12:24:00 PM, Anonymous Anonymous said...

By the way - as far as the stadium subsidy thing goes which I mentioned in #6 above - the confusion between subsidies for building new stadia and using existing ones is present in Bradbury's book. So Zimbalist's comment is fair. [And how can mere use of a public stadium itself be a subsidy? The USFL teams paid rent; wouldn't you have to know the details of those leases to know if any "subsidies" were involved? As far as I'm concerned, Bradbury has not made his case at all on this point.]

Tango,
I think it is premature to call Zimbalist unpliable based on this alone, until we see his response to Bradbury's response. Will he admit the errors he did make, or not?
I think the intereresting point to be taken from Zimbalist's review is not which of them is weaker on sabermetrics, but that Zimbalist seems to think that Bradbury has made several errors in economic and econometric analysis. I can't fully judge, but I do think Zimbalist has exposed some sloppy arguments. [I see Guy has just posted a rather similar point.]

Phil,
you usually write with such clarity on these topics that I must apologize in advance for disagreeing, in case I am being monumentally stupid. I don't see how you can say that Zimbalist is wrong to criticise Bradbury's interpretation of the coefficients and point to elasticity instead. At most he is wrong on the 2nd idea, and that is not clear to me. [n.b. Z is rejecting Bradbury's interpretation of B's coefficients from B's equation, not necessarily also rejecting any interpretation of coefficients from a better constructed equation.]

I've also followed the discussion at BTF and I can't quite understand what you're saying here and there. At BTF, in post #11 [reiterated in #30], Walt Davis mentions that collinearity between OBP and SLG makes the interpretation of the regression coefficients problematic, and he concludes "Whatever these coefficients might tell you, they don't tell you the relative impact of BA/OBP/SLG on scoring."
Now that does seem to be exactly what Bradbury has done, though he reports both the ratio between the coefficients (approaching 3, he says, when his data is restricted to AL 1998-2002 to match the period covered by Moneyball), and the elasticity, which he reports as 1.43 for 1998-2004 [ doesn't provide AL 98-02 separately]. About elasticity he says [TBE p160]: "Elasticity is also a marginal impact measure, but it interprets the effect in percentage terms at the average value for runs per game and the statistic. For example the elasticity of 1.22 percent for NL OBP when all of the metrics are included means that a 1 percent increase in OBP is associated with a 1.22 percent increase in runs per game. This metric is useful for comparing the sensitivity of the impacts in the same terms."

Bradbury and Zimbalist are using the first alternative meaning for "elasticity" discussed by "Mr Man" at BTF post #25. In post #27 there you say that if Zimbalist means that you stand by your original post. In post #32 you suggest you agree with Davis, but while he doesn't say directly whether elasticity is the appropriate measure, nowhere does he back away from his critique in #11 about the interpretation of Bradbury's coefficients. Instead you and he appear to have gone on a tangent, discussing alternative regression equations (no longer Bradbury's) in which interpretaion of co-efficients perhaps would be straightforward.

Now I don't know offhand if there is a difference or not between the ratio of the coefficients approach (if collinearity is eliminated) and elasticity. They may give the the same answer in this case, and conceivably they resolve to the same meaning if the math is followed through. I just point out that elasticity does seem to be in the running to be an acceptable measure, according to both Zimbalist and Bradbury. I hope you can explain as clearly as you usually do why you think the elasticity measure is obviously to be rejected.

regards
Joe

 
At Wednesday, December 12, 2007 12:38:00 PM, Blogger Phil Birnbaum said...

Hi, Joe,

Actually, I am trying hard to avoid commenting at all on whether elasticity is a good thing to look at or not. What I'm trying to say is this: since Moneyball came out, there has been a debate on what the relative value of the coefficients should be. JC weighed into that debate. Zimbalist now criticizes JC for caring what the coefficients should be, and suggests elasticity instead.

This is what I am criticizing Zimbalist for. He is not aware that the coefficient question is an outstanding controversy in sabermetrics, and accuses Bradbury of concentrating on the wrong point. This is not fair.

(Over at BTF, I made a couple of comments on the actual regression, which perhaps I shouldn't; those comments have nothing to do with the Zimbalist point. They were technical comments on what the coefficients mean in JC's regression.)

Now, if you're saying that the elasticity calculation necessarily gives you the same answer as the coefficient question, I'm OK on that.

Actually, now that I think about it, perhaps it is. At the margin it is -- it gives the right answer for what an *additional* point of OBP is worth compared to an *additional* point of SLG. Does that mean that it works "all the way down" so that EVERY point of OBP is worth 1.8 of EVERY point of SLG? I bet you're right, that it does.

So perhaps Zimbalist was correct that the elasticity ratio can be used to figure this out. However, on page 161 (thanks for the page reference, I couldn't find it), in Table 24, JC gives the results for a regression on OBP and SLG alone. The results are in the 1.9 range.

Having said all that: Zimbalist's arguments immediately preceding that one sentence about elasticity are irrelevant. The fact that SLG is normally larger than OBP does NOT have any bearing on the issue at all. And the problem with Bradbury's 3.0 is NOT that the numbers are of different magnitudes, but simply that AVG is included in his regression.

Basically, it seems like Zimbalist used Bradbury's data to get the right answer, but didn't understand (or properly criticize) Bradbury's argument that gave him the wrong answer.

Does that help? Thanks for making me think about this more, I see where you're coming from now.

 
At Wednesday, December 12, 2007 12:53:00 PM, Blogger Phil Birnbaum said...

Guy (two posts up),

Agreed! I don't mean to imply too much about the quality of Bradbury's work (either good or bad). Zimbalist's review of the sabermetrics falls on its own, independent of whether Bradbury is mostly right or mostly wrong about things.

You seem to be saying that if Zimbalist says the sabermetric parts of JC's book (which we agree are flawed) are the best, then, geez, the finance/economics parts must definitely need improvement, especially considering Zimbalist is an expert on that stuff. And you may be right -- I'm not disputing that at all.

I suppose I could thoroughly go through the non-sabermetric parts of Zimbalist's criticism, and Bradbury's response, and look up the original texts and see which side I fall on. But because I haven't done that yet, I'm trying to deliberately avoid commenting on those other parts of the review, and I don't mean to imply ANYTHING about those, because I haven't done the work yet, or given it much thought.

My overall thought is this: "Geez, Zimbalist really didn't understand the sabermetrics, got that part of the review wrong, and unfairly maligned parts of Bradbury's work. I wonder if he did the same for the finance parts? You could argue that most of those criticisms are probably valid, because Zimbalist is an expert there, but I should check before I open my mouth. And I should give Bradbury some benefit of the doubt, because Zimbalist was so confidently wrong on the other stuff."

 
At Wednesday, December 12, 2007 1:01:00 PM, Anonymous Anonymous said...

"You seem to be saying that if Zimbalist says the sabermetric parts of JC's book (which we agree are flawed) are the best, then, geez, the finance/economics parts must definitely need improvement, especially considering Zimbalist is an expert on that stuff."

No, I didn't mean that. I'm not qualified to judge the real finance/economics part of JC's work, or Zimbalist's critique (except insofar as it incorporates sabermetric ideas like player valuation). I just meant that IF Zimbalist were right about the sabermetric chapters being stronger, then that would indeed be a severe indictment of the finance chapters.

FYI: Zimbalist has now posted a short reply to JCB.

 
At Thursday, December 13, 2007 9:56:00 PM, Anonymous Anonymous said...

I fiddled with the math and found an additional discussion of elasticity; contra my speculation, it is not mathematically equivalent to the ratio of coefficients. It is a method of "normalizing" coefficients when units of each variable may be on a different scale. Found this in R.Berk, "Regression Analysis" [2004] pp 29:
"If the units in which y and/or x as recorded have little intuitive meaning, some data analysts feel that they can better understand the size and the meaning of the slope if it can be interpreted in either percentage units or standard deviation units." [The first is elasticity, the second, the beta coefficient]. I think the reason Zimbalist quoted the difference in scale between OBP and SLG was because he was justifying the choice of elasticity for interpreting the results of the regression.

I don't agree that the sabermetric debate has been about the size of the coefficient itself either, so in my view it is not fair to criticize Zimbalist for not "knowing" that. I think the question posed has been what is the value of a point of OBP, relative to a point of slugging pct, and non-regression answers have been given to that question by Tango, Guy and Mark Pankin at least. In his book, Bradbury posed the question as how many more times valuable a point of OBP than a point of SLG, and he implicitly endorsed the correctness of the "moneyball" answer of 3x for 1998-2002.

In sum, Bradbury seemed to endorse what I think we agree to be an incorrect answer to the question as I've described it; Zimbalist criticized it and gave an answer we agree to be approximately correct; we don't really understand whether Zimbalist's reasoning in doing so was sound or not.

Even if you are right and I am wrong about what the sabermetric question really is, Zimbalist was reviewing Bradbury's text. Given the question as Bradbury framed it, I think the fairness of Zimbalist's criticism of Bradbury really does just depend on whether he is right or wrong on the correct way to "value" the coefficients.
regards
Joe

 
At Thursday, December 13, 2007 10:53:00 PM, Blogger Phil Birnbaum said...

Joe,

I think the ratio of the coefficients is the same as the relative value of a point of each, isn't it? In which case, the relative size is irrelevant.

I still do think that Zimbalist didn't understand the question, but I do concede that he did give the right answer ... and I shoudl consider that you and others think what he did was OK. We might have to agree to disagree, but I'll think about what you're saying a bit more.

 
At Friday, December 14, 2007 10:30:00 AM, Anonymous Anonymous said...

Phil,
you may find it helpful to read Bradbury's remarks in The Baseball Economist on pp 170-71, in which he appears to prefer elasticity when it comes to evaluating the impacts of the variables he used for his regression analysis of pitching performance. He gives a brief explanation there.

 

Post a Comment

<< Home