Sabermetric Research

Do NHL teams get a boost after killing a two-man advantage?

2012-01-30T10:34:00.008-05:00

In an OHL game I was watching the other day, one of the teams had a two-man advantage and didn't score. The announcer was disappointed that the shorthanded team to get a boost from having killed off the penalties, as conventional wisdom says they should.

Is conventional wisdom right? Now that I have access to a database of NHL games (thanks again to the Hockey Summary Project), I was able to check.

This study is basically the same format as the study I did on fights a few weeks back. I found all games from 1967-68 to 1984-85 where one team killed off a two-man advantage (of any length). Then, I found a random control game, which matched the score differential and the relative quality of the home and road teams. When I was done, I had two pools, each comprised of 1,703 games.

The teams that killed the penalties scored an average 0.26 more goals than their opponents from that point to the end of the game (actually, to the 17:00 mark of the third period). On the other hand, the control team scored only 0.12 more goals then their opponents.

That's statistically significant, at almost exactly 2 SDs.

I'll put that in chart form to make it easier to read, along with the SD. I use the term "killing teams" to mean the ones that actually killed off the two-man advantage.

Killing teams .... +0.26 goals (+/- 0.05)
Control teams .... +0.12 goals (+/- 0.05)
------------------------------------------
Difference ....... +0.14 goals (+/- 0.07)

At six goals per win, you'd have expected the extra goals to have resulted in around 40 extra wins. They actually resulted in 32 extra wins. Actually, 36 extra wins, minus 8 fewer ties:

Killing teams .... 836-604-263
Control teams .... 806-626-271
------------------------------------
Difference ....... +36 wins, -8 ties

So, should we conclude that killing off a two-man advantage causes a psychological boost? Well, not so fast. Because, after you take two consecutive penalties, the referee is very likely to try to even things up by giving future penalties to the other team.

The difference of +0.14 goals is almost exactly what you'd get from a single power play. So, if the result of surviving a two-man advantage is that you get one extra "free" power play in the remainder of the game, that would explain the results exactly.

As it turns out, it's not quite that high. It's only half that high. On average, the teams that survived being shorthanded two men got about half an extra power play in the remainder of the game:

Killing teams ... +.346 power plays rest of game
Control teams ... -.130 power plays rest of game
------------------------------------------------
Difference ...... +.476 power plays rest of game

That leaves about 0.07 goals per game as the unexplained difference. It's only 1 SD, which is no longer statistically significant. It's about the effect of half a power play. Or, with an average save percentage of .900, it works out to 7/10 of an additional shot on goal.

--------

We can also handle the penalty issue another way. We can insist that when we choose a control game for the real game, we make sure the control team was the lone who took the last penalty. That way, we'd expect some of the referee "evening up" difference to disappear. Perhaps not all of it, because a two-man advantage isn't the same as a one-man advantage -- but at least part of it.

The additional restriction reduced the sample size to 1,662 games; for the remaining 41 games, I couldn't find a suitable control.

As it turns out, the goal difference stays about the same, even though the penalty difference is significantly reduced:

Killing teams ... +0.25 goals (+/- 0.05)
Control teams ... +0.08 goals (+/- 0.05)
----------------------------------------
Difference ...... +0.17 goals (+/- 0.07)

Killing teams ... +.340 power plays rest of game
Control teams ... +.032 power plays rest of game
------------------------------------------------
Difference ...... +.308 power plays rest of game

The difference of .308 power plays accounts for around .04 goals of the observed .17 difference. That leaves .13, which is a little less than 2 SD from zero. Not statistically significant, but close. (Technically, it's even less than that, because the control games aren't completely independent. Also, when I ran the study a second time, I got +0.10 goals instead of +0.08, which lowers the difference. So think of the 1.9 SD as probably a bit too high.)

Strangely, though, there wasn't as much difference in game results; only the equivalent of 13.5 wins:

Killing teams ... 815-591-256
Control teams ... 807-610-245
------------------------------
Difference: +8 wins, +11 ties

Again at six goals per win, you'd expect 47 wins, not 13.5. What happened?

Well, it turns out that the "killing" teams spent a lot of their goals winning blowouts. For instance, in games won by six goals or more, they were 81-34. The control group was only 73-51.

In those games, the difference was 12.5 wins. That normally "costs" 75 goals, but, for these games, the difference was really around 150 goals. So, that accounts for 75 of the 282 goal difference right there.

The "killing" group also "wasted" goals in the 3- and 4-goal games. That was offset by the opposite effect in five-goal games, but not by much.

------

If you recall, we found the same effect when we looked at fighting: teams that started a fight appeared to score more goals, but not necessarily win more games.

What connects the two studies is ... penalties. It could be that teams that get penalized a lot win a lot of blowouts. Not necessarily because of cause-and-effect, but because it just so happened that, between 1967 and 1984, certain teams just happened to be high in both categories.

Or, it could be coincidence. Or, it could be something else.

------

For my bottom line, I'd say: after killing off a two-man advantage, teams did appear to benefit by about 1/7 of a goal. Half of that can be traced to referees calling fewer penalties against them in the remainder of the game.

The other half is unknown. It's not statistically significant, so you have to give serious consideration to the idea that it's just coincidence ... but the teams *did* appear to benefit, by around 0.07 goals.

Historically, the average size of the "boost" in a team's play after a two-man kill has been small: the equivalent of less than a single shot on goal over the remainder of the game.

GiveWell: Overcomplicating research studies can cost lives

2012-01-20T13:02:00.008-05:00

"GiveWell" is an organization that evaluates charities. Not just the usual things -- how well they're run, or how much money goes to administrative expenses -- but also how much good they do for the money they receive.

The idea is: if you have $100 to give to try to make the world a better place, shouldn't you give that $100 where it would give the most benefit? Not just to whoever shows up at your door that day, or whatever organization makes you feel guiltiest, or whoever's suffering kids look the cutest ... but, seriously, to where you can do the most good.

That might not appeal to everyone. If you donate to maximize your own good feelings, instead of the good your donation actually does, GiveWell's evaluations won't make much difference to you. Some people hate to say "no", and so they prefer to give $5 to each of the twenty charities that ask for money. Some people prefer to give to diseases that killed their loved ones, or diseases associated with heroes like Terry Fox. Some people give to causes that signal their political views. Most people prefer to give to help people in their own city or country, even when their dollars will save many more lives abroad.

(I've done all these things, and I'm bit embarrassed about some of them. But I'm not alone. I mean, people give money to the Children's Wish Foundation to send a terminally ill kid to Disneyland ... which is nice, but, that same amount of money might actually save ten lives if they sent it to Africa where kids are actually dying of things that are easily preventable. I'm not sure what's up with me, and my fellow humans, sometimes. But I digress.)

So, in at least one sense, GiveWell is to donors what sabermetrics is to Joe Morgan. It does analysis to reach conclusions that some might find uncomfortable.

However, in another sense, what GiveWell does is *unlike* sabermetrics, in that it usually doesn't try to get down to the third decimal place. It argues that it can evaluate charities heuristically, that the differences are big enough that they can figure out which charities are the best, using the charities' own reports. As I interpret what they're saying, GiveWell can very easily tell you whether a charity is a Danny Ainge or an Albert Pujols, and it can even tell you more subtle things, like whether a charity is a Joe Carter or an Albert Pujols. But it doesn't try to figure out if a charity is a Ryan Braun or an Albert Pujols. It will just tell you that both are recommended.

That is, GiveWell argues that its goals are better met by the transparency of its recommendations than by any detailed, opaque analyses.

Which is almost exactly what I argued in one of my recent posts -- that, in research, simplicity and transparency are more important than rigor. Simple studies make it much easier to understand the results and catch the inevitable errors. A gentleman from GiveWell, Elie Hassenfeld, read that post, and pointed me to a particular example of a serious error that his organization uncovered.

(Disclaimer: I don't really know much about GiveWell. However, I've been impressed by what I've seen, and at least two of the blogs I read and respect (here's one) say very good things about them. So my Bayesian evaluation of them is quite high.)

-----

As I said, GiveWell doesn't believe they need detailed statistical cost/benefit studies to decide which charities to recommend. However, charities themselves often use such analyses to decide where the money should be spent. There's a whole bunch of organizations and academics devoted to figuring out how to save the most lives for the fewest dollars.

With that objective, the Bill and Melinda Gates Foundation donated $3.5 million to fund a study, "Disease Control Priorities in Developing Countries". They published a report ranking various interventions on cost-effectiveness. The Gates Foundation didn't do that itself -- it was done jointly by The World Bank, the National Institutes of Health, the World Health Organization, and the Population Reference Bureau. Those sound like heavyweights in the world health field.

The results found that -- unsurprisingly to me -- hygiene promotion was the cheapest way to reduce death and disease. The second cheapest, though, was deworming. Specifically, "soil-transmitted helminth" (STH) deworming treatments.

After the report was released, the Gates Foundation provided another $4.4 million to promote the findings. And the findings did indeed attract serious attention. GiveWell writes,

The DCP2’s cost-effectiveness estimates for deworming have been cited widely to advocate a greater focus on treating STH infections, including in:

-- an article in The Lancet

-- a report by REACH, a consortium of large international NGOs and other organizations working to end child hunger, which labeled deworming one of 11 “promoted interventions”

-- the most-cited paper published in the journal International Health

-- an editorial by Peter Hotez, a co-founder of the Global Network for Neglected Tropical Diseases, which has received more than $40 million in funding from the Gates Foundation

-- work by charity evaluators, such as GiveWell, Giving What We Can, and the University of Pennsylvania’s Center for High Impact Philanthropy.

But, as GiveWell later discovered, it turns out the STH estimate was wrong.

That doesn't sound too serious, but here's the thing: it's not just that the estimate was wrong. It was wrong by a factor of almost ONE HUNDRED. The study said that you could save one "disability-adjusted life year" by spending $3.41 on deworming treatments. But, after correcting for the (acknowledged) errors in the study, the actual number was $326.43.

All these well-respected organizations, with serious researchers and serious money, wound up promoting a conclusion that was about as wrong as it could have been. Until the error was caught, then, effectively, 99% of the money devoted to STH treatment was wasted.

How did GiveWell catch the error? Subject matter expertise, mostly. In reading the report, they noticed that the STH estimate was much, much lower than other estimates they had seen. Instead of just assuming that this research was somehow better than the previous studies, they investigated.

That seems like just common sense, right? If you see a study that says an iPod can be bought for $3, when you know it usually costs $300, you should look again, shouldn't you? But that didn't happen until someone at GiveWell decided to figure out what was going on.

So they wrote to one researcher, who sent them to other researchers, who sent them complicated spreadsheets. They tried to figure those out, but they couldn't, so they wrote back and forth with questions and explanations. They were referred to still another researcher, who sent them a copy of yet another study that was the source of some of the data.

Eventually, they figured out where the issues were ... if you want a full explanation, it's in their post. It was a lot of detailed, technical effort to figure out what went wrong, and which parameters were in error.

GiveWell's conclusions:

We believe that the errors we’ve found in the estimate would have been caught by a helminth expert independently examining the estimate. Therefore, the presence of these errors implies to us that there has been no such examination. If this is the case, it would argue against the reliability of the DCP2’s estimates in general.

We’ve previously argued for a limited role for cost-effectiveness estimates; we now think that the appropriate role may be even more limited, at least for opaque estimates (e.g., estimates published without the details necessary for others to independently examine them) like the DCP2’s.

More generally, we see this case as a general argument for expecting transparency, rather than taking recommendations on trust - no matter how pedigreed the people making the recommendations. Note that the DCP2 was published by the Disease Control Priorities Project, a joint enterprise of The World Bank, the National Institutes of Health, the World Health Organization, and the Population Reference Bureau, which was funded primarily by a $3.5 million grant from the Gates Foundation. The DCP2 chapter on helminth infections, which contains the $3.41/DALY estimate, has 18 authors, including many of the world’s foremost experts on soil-transmitted helminths.

Absolutely right. You can't substitute credentials for subject matter expertise, and you can't substitute complexity for transparency.

And, one thing I would add: when a study appears to discover that you can get benefits at 99% off the original, well-accepted price ... you have to be suspicious about accepting that conclusion, even if you have no other reason to believe there was any mistake.

-----

P.S. GiveWell expands on the theme here.

Are more NHL penalties called in back-to-back games?

2012-01-15T00:15:00.006-05:00

In a comment to one of the posts on "make-up" penalties, J.-P. Martel wrote,

"... blow-outs can easily lead to situations that get out of hand, so referees may call penalties on the leading team so that the trailing team still thinks it has a chance to come back, rather than resort to fighting to "prepare" the next game between the two teams.

Actually, you may want to check penalties in the second half of the third period when the teams' next game is (or may be, depending on outcome) against each other (particularly in the playoffs), as opposed to when it's not."

So I did. And, J.-P. is right, it looks like there's something there.

I found all cases from 1967-68 to 1984-85 where teams played back-to-back games (regular season only). Then, I formed three groups:

-- first game of back-to-back games
-- second game of back-to-back games
-- other games that year between those two teams

It turns out that, overall, there are more penalties than usual in the first game, and fewer penalties than usual in the second game:

First game .... 12.36
Second game ... 10.87
Other games ... 11.77

Broken down by periods:

-------------- Gm 1 --- Gm 2 --- Other
--------------------------------------
Period 1 ..... 4.78 ... 3.98 ... 4.37
Period 2 ..... 4.25 ... 3.75 ... 4.12
Period 3 ..... 3.32 ... 3.13 ... 3.26
--------------------------------------
Total ....... 12.36 .. 10.87 .. 11.77

So: there's 0.6 extra penalties in the first game, and 0.9 fewer penalties in the second game.

I thought the second game would be dirtier because the player are holding recent grudges from the previous game, but the numbers show the opposite. The players seem to be more aggressive early, rather than late. In fact, more than half the "first game" effect happens in the first period. By contrast, a large "second game" effect seems to last two periods rather than one.

Most of the differences are statistically significant, which suggests that they're all real. For those scoring at home, here are the standard errors:

-------------- Gm 1 --- Gm 2 --- Other
--------------------------------------
Period 1 ..... 0.16 ... 0.12 ... 0.06
Period 2 ..... 0.13 ... 0.11 ... 0.06
Period 3 ..... 0.15 ... 0.12 ... 0.07
--------------------------------------
Total ........ 0.30 ... 0.24 ... 0.13

Finally, coming back to J.-P.'s hypothesis about the second half of the third period of the first game, here are the numbers:

First game .... 1.76
Second game ... 1.63
Other games ... 1.67

So, yes, there's a small effect where, when the teams are going to meet again next game, the referee calls more penalties than normal in the last ten minutes of the third period. Whether that's because of the referee, or the players, we can't tell.

Taken alone, these differences aren't statistically significant. But, considering they match the pattern, and the broader picture is statistically significant, we can be fairly confident that this is a real effect we're seeing.

That's actually why I saved J.-P.'s scenario for last, so I could first show that the effect is probably real and not just random.

-----

UPDATE, 1/15/2012:

Technical note: the "other games" rows and columns are weighted by games, rather than matchups. Suppose teams A and B had back-to-back games, and so did C and D. But A and B met only 2 other times that year, while C and D met 4 other times. That means that C/D will be overrepresented in the "other games" column.

If I reweight that column so A/B and C/D get equal weight, the results change just a little bit. These are the revised "other" columns:

Overall ...... 11.48 (was 11.77)
1st period .... 4.24 (was 4.37)
2nd period .... 4.07 (was 4.12)
3rd period .... 3.15 (was 3.26)
Last 10 min ... 1.59 (was 1.67)

Do hockey fights lift a team's performance? Part II

2012-01-13T10:46:00.005-05:00

The previous post was a study on NHL fights. It found that, generally, a fight doesn't help the team that it's sometimes said to help (the team that's behind in the game, for instance), but in one particular case, MAYBE it did. That was the case where:

(a) one team was behind in the game
(b) that team fought more regularly than the other team, and
(c) the player fighting also fought more often than the other team's fighter.

In that situation, that team appeared to benefit by around 0.13 goals, as compared to a similar team that didn't fight. That was about the same as one extra power play.

However, the result was not statistically significant, being only 1 SD away from zero. Still, I left it at least a little bit open whether the effect *might* be real.

Tom Tango is more skeptical than that:

It’s not monkeys at a typewriter creating Shakespeare, but it’s close.

Well, I have some more evidence that supports that point of view.

I repeated the study 27 times, to get a larger sample of random control games. (I didn't pick the number 27 beforehand; I just ran the thing over and over until I got sick of it.) Here's the average of those 27 runs:

Actual teams .... -0.18 goals
Control teams ... -0.29 goals
------------------------------
Difference ...... +0.11 goals

To remind you what this means: the fighting team meeting the conditions was outscored by its opponent by 0.18 goals over the rest of the game. On the other hand, the control teams, which were selected randomly from games which matched as closely as possible (except for the fight), got outscored by 0.29 goals.

So, it looks like the team that fought gained 0.11 goals per game. As I said, that result is not statistically significant.

But now, here's the new thing. Even though the fighting team gained 0.11 goals, it actually lost more games. Here are the records, in W-L-T format:

Actual teams .... 52-274-38
Control teams ... 49-267-48
----------------------------
Difference ...... -2 wins

So, even though the fighting teams did better on the scoreboard, they did worse in terms of winning games. Actually, they won three extra games, but they lost seven more and tied 10 fewer. That adds up to minus four points in the standings, which is why I write "-2 wins". (I'm ignoring the "pity point" for an overtime loss.)

You wouldn't expect this to happen, that you score more goals but lose more games. The better your goal differential, the better your outcomes should be. I think I saw Gabriel Desjardins write, somewhere, that six goals equals one win. The observed difference of +0.11 goals per game, over 364 games, equals around 40 goals, which is almost seven wins.

But instead of winning seven extra games, the fighting teams *lost* two extra games.

Why did this happen? I think it's just luck, well within the bounds of random error. I think the +0.11 goals per game is random chance, I think the -2 wins is random chance, and I think the discrepancy between the two results is also random chance.

In any case, if you don't like all this talk of significance levels and randomness, you can just summarize like this: overall, the teams that fought wound up very slightly better on the scoreboard, but very slightly worse in the standings.

Do hockey fights lift a team's performance?

2012-01-10T11:59:00.011-05:00

It's been said that when an NHL team needs a lift, a fight can jolt it out of its complacency and make it better. And, just a few days ago, the media cited a study by researcher Terry Appleby, of powerscouthockey.com, showing that momentum (in terms of shots on goal) usually increases for at least one team after a fight.

But, if *either* team can benefit from a fight, what's the point? You want to know if *your* team can benefit from a fight, at least more than the other team does.

The problem is: how can you know that? A fight involves both teams, so if it helps one, it hurts another by the same amount. If you look at both teams, you'll always find the total effect to be zero.

So, the "fighting helps a team" theory has to say *which* team is helped. The most logical interpretation would be that that the fight helps the team that instigated it.

If you're going to study that, you need to know which team is the instigating team. That's tough to figure out from historical data. But, one shortcut would be to assume that the team that generally gets involved in more fights is the team that's more likely to have instigated. The 1974-75 Philadelphia Flyers took 76 fighting penalties (actually, 76 offsetting majors, which I used as a proxy for fights). That same season, the expansion Kansas City Scouts took only 19. It seems fair to assume that if a fight broke out at a Flyers/Scouts game, it was the Flyers who were likely responsible.

On that assumption, I decided to check.

Using data from the Hockey Summary Project, I looked at fights between 1967-68 and 1984-85, and checked to see how the more-likely-to-fight team did in the remainder of the game. Then, I found a control game to match it with. The result was two large groups of games, which could then be compared.

I'll give you an example of how the controls were found.

On Feburary 16, 1969, the Bruins played the Black Hawks at Chicago Stadium. Just as the first period ended, with the score 2-0 Chicago, the Bruins' Don Awrey got into a fight against Stan Mikita of the Hawks.

I looked for a game to serve as the control for that Boston/Chicago game. What I wanted was:

1. A game in the same season, the season before, or the season after;
2. ... where the home team had the same size lead at that same time of the game;
3. ... and where the two teams were of roughly similar relative quality.

#1 and #2 were non-negotiable (except that all differences of 4 or more goals were considered the same). But, for #3, the quality only had to be close, within two goals (which I'll explain in a minute).

I started pulling random games until I found one that matched all three requirements. In this particular case, the control wound up being the Bruins vs. Rangers game of February 23, 1969.

That game qualifies under the rules because

1. 1968-69 is in the same season as the original;
2. That game had the home team also leading by two goals at 20:00 of the first period, and
3. The two sets of teams are of similar relative quality.

Now, let me explain #3.

In 1968-69, the Bruins were +82 in goal differential (303 goals for, 221 against). The Black Hawks were +34 (280-246). So, for the original game, the home team was 48 goals worse than the visiting team.

Since the control game was the same year, the Bruins were still +82. The Rangers were +35 (231-196). So, in the control game, the home team was 47 goals worse than the visiting team.

Since "47 goals worse" is within two goals of "48 goals worse," that's close enough for the Bruins/Rangers game to serve as a control. If it hadn't been within two goals -- which is most of the time -- that game wouldn't have qualified under #3, and I would have tried another random game. (If there were absolutely no games that qualified under #3, I would have taken the one where the team quality was closest in goals. If none of the random games had qualified under #1 and #2, I would have thrown the original game out of the study -- but that never happened.)

OK, so now we have our real game, and our control game.

Which team in our real game are we going to expect to have gotten the "lift" from the fight? In 1968-69, the Bruins had 41 fights, but the Black Hawks had only 20. So, the assumption is that the fight was more the work of the Bruins, and they should be the ones expected to benefit.

How did the Bruins do in the rest of the game relative to the Black Hawks?

Well, the final score was 5-1 Hawks. Since it was 2-0 at the time of the fight, that means the Black Hawks outscored the Bruins 3-0 in the remainder of the game. In other words, a "minus 3" goal differential for the visiting Bruins. (I excluded any goals in the last three minutes of the third period, to make sure empty-net goals didn't screw things up.)

What about the control game? That game actually wound up 9-0 Rangers, which means 7-0 Rangers from the fight to the end of the game. Since the "real" game was relative to the Bruins, the visiting team, we also want to express the control game from the standpoint of the visiting team. So that's "minus 7".

So, our score so far is:

Actual games: -3.0 goal differential for the fighting team

Control group: -7.0 goal differential for the control team

So far, it looks like fighting helps, by four goals a fight!

Of course, that's only one game. I repeated this process for every fight from 1967-68 to 1984-85. Actually, not *every* fight. First, I included only fights where one team appeared to be significantly more aggressive than the other (specifically, where the two teams were 10 or more fighting penalties apart for the season). Second, I included only first- or second-period fights, to increase the amount of time for the "lift" effect to make itself felt.

Even with those restrictions, there were 2,834 fights total. The results:

Fighting teams ... -0.04 goals
Control group .... -0.02 goals

The team with more fights was 0.04 goals worse than the other team over the remainder of the game. It "should have" been 0.02 goals worse. (Both numbers are negative probably because the teams that got in more fights were slightly worse teams overall than their opponents.)

So, there seems to be a small, negative effect: a team loses one additional goal for every 50 fights. But, that difference isn't even close to statistically significant. It's less than one SD from zero. (The two individual SDs are about 0.04 each, so the SD of the difference is around 0.06.)

Conclusion: it doesn't appear that fighting helps a team.

-----

Maybe a difference of 10 fights a year isn't enough to separate the two teams? I redid the study, but required the teams to be 20 fighting penalties apart. That reduced the sample size to 1,581 each group. The results were about the same (the +/- in parentheses is the standard error):

Fighting teams .... 0.00 goals (+/- 0.05)
Control group .... -0.03 goals (+/- 0.06)

-----

Looking at the entire database, I found that the average fight starts with a goal differential of 1.617. The average goal differential in all other games, weighted by the times of fights, is 1.421 goals. So, it seems like fights start when the game is a little more lopsided than usual.

So, maybe it's the team that's *trailing* that starts the fight, in an effort to wake itself up. Maybe we should look at trailing teams, not goonier teams.

I tried that. I threw away all situations where the score was tied when the fight happened, and looked at all the rest. The results:

2,941 datapoints
----------------
Trailing teams ... -0.20 goals (+/- 0.04)
Control group .... -0.19 goals (+/- 0.04)

Again, no real difference.

-----

Trying again, but looking only at fights where one team was trailing by at least three goals:

591 datapoints
--------------
Trailing teams ... -0.25 goals (+/- 0.08)
Control group .... -0.29 goals (+/- 0.08)

Nothing there, either.

-----

Is it possible that the benefit accrues only to GOOD teams trailing by three goals? Those are the teams playing the worst relative to their abilities, so the "wake up" effect should be strongest. Here are teams trailing by 3 goals that were at least +30 in goal differential for the season:

122 datapoints
--------------
Trailing teams ... +0.14 goals (+/- 0.17)
Control group .... +0.17 goals (+/- 0.16)

Nope. What if we look at good teams trailing by *any* number of goals?

841 datapoints
--------------
Trailing teams ... +0.40 goals (+/- 0.07)
Control group .... +0.26 goals (+/- 0.07)

Aha! This time, there's a small "lift" effect, at about 1.4 SD. But, why would there be an effect for teams trailing by 1 goal, but not for teams traling by 3 goals?

I got curious and ran the same study again, and this time the random control group came in at +0.33, bring the difference down to 1.0 SD. (Of course, it's not appropriate to dismiss the first result just because the second one came out less extreme.)

-----

At this point, you might reasonably argue that the rules "team with more majors that year" and "team trailing in the game" are not precise enough in selecting teams that started the fight. So, this time, I assumed the fight was started by the *player* with the most majors that season, rather than the *team* with the most majors that season. So when the goon of a pacifist team starts a fight with a pacifist of a goon team, you go with the goon player on the pacifist team. The results:

4,185 datapoints
----------------
Goonier Player ... +0.01 goals (+/- 0.03)
Control Group .... -0.02 goals (+/- 0.03)

Again, less than 1 SD difference. There's not much difference between this "goon player" breakdown and the previous "goon team" breakdown, probably because most of the goonier players also played on goonier teams. But it was worth a try.

-----

Finally, one last try. For this run, I combined all three criteria. To be included in the study:

(a) one team had to have at least 20 more majors for the season than its opponent;
(b) that same team's fighter had to have more majors that year than his opponent; and
(c) that same team had to be trailing in the game.

This *has* to work, right? I mean, that pushes all the right buttons: a truculent team, with a figher selected for that purpose, behind in the game and likely to be needing a lift. If *those* teams don't benefit from the fight, then who would?

I expected the same non-result, but, this time, we get the biggest effect so far:

364 datapoints
--------------
Teams qualifying ... -0.18 goals (+/- 0.10)
Control group ...... -0.38 goals (+/- 0.11)

There's a difference of .20 goals -- almost a fifth of a goal per fight! Taken at face value, that means that when a team like that starts a fight, it benefits by even more than a power play (which has a 15 to 20 percent success rate).

That difference is still only about 1.4 SD from zero. Still, I hate to just dismiss it. I've always thought that if you get a result that's significant in the real-world (hockey) sense, but it's not statistically signficant, that's a problem with your study -- it's just that you haven't used enough data to be able to prove anything. We should still be open to the possibility that the effect might be real.

I ran it a few more times, to check if maybe the control group was just a random outlier. The extra results:

Control group: -0.26 goals
Control group: -0.21 goals
Control group: -0.27 goals
Control group: -0.38 goals (again)
Control group: -0.38 goals (again)
Control group: -0.29 goals

So, the original run was a little extreme, but not much.

There are, however, some mitigating factors. First, the control group numbers aren't all independent, since there's a limited number of control games to choose randomly from. Second, we obviously can't do extra runs to reduce random chance in the *real* games, but it's still possible those teams scored more goals for random reasons having nothing to do with any lift they got from the fight. Third, the SDs of both groups are a bit understated: I calculated them based on the assumption that games are independent, but they're not -- a real game appears in the study multiple times, once for each fight, and a control game could get randomly selected more than once, too.

If you average the seven control groups in the seven repetitions of the study, you get -0.31 goals. That's 0.13 goals worse than the actual games. Taking into account the fact that we ran the control group five times, the 0.13 difference is now around 1 SD.

Oh, and this is as good a time as any to emphasize that I could also have screwed up somewhere ... I've already had to rerun everything once when I found a misplaced parenthesis in my code.

-----

So, I guess, our overall conclusion from this study isn't completely certain. We wind up with a summary like:

1. The effect doesn't seem to exist for run-of-the-mill fights.
2. When a goon fighter on a goon team fights when his team is down, it seems to benefit that team by 1/8 of a goal, or a bit less than a normal power play.
3. But, that effect isn't statistically significant, so we have some doubts that it's real.
4. And, with only 364 such datapoints qualifying out of around 5,000, only a small percentage of fights match the criterion for that kind of boost.

If you had to reduce that to one line, it might be:

At best, there might be a small effect in certain specific circumstances ... but much, much less than sportscasters make it out to be.

UPDATE: Part 2 is here.

Do NHL referees call "make up" penalties? Part IV

2012-01-09T17:11:00.004-05:00

A couple of links to other similar studies on penalty-calling:

1. Commenter Jack linked to this article with some basketball foul-calling data. Turns out the more consecutive fouls against one team, the more likely the next will go to the other team.

2. Another reader pointed me to a 2009 hockey study (web version here, PDF here) by Jack Brimberg and William J. Hurley. They looked at the first three penalties of every game, and found results similar to what I found.

Ken Dryden tracers from "The Game"

2012-01-07T12:58:00.007-05:00

In my review of Ken Dryden's book "The Game," I listed seven of the details that I tried to trace. I now have an eighth one, and then a retrace of the second one.

These two updates were originally posted to the SIHR mailing list. The original seven are here. Thanks again to the Hockey Summary Project for the data making these tracers possible.

----

8. Here's Dryden, from page 121 of my edition:

"A few months ago, we played the Colorado Rockies at the Forum. Early in the game, I missed an easy shot from the blueline, and a little unnerved, for the next fifty minutes I juggled long shots, and allowed big rebounds for three additional goals. After each Rockies goal, the team would put on a brief spurt and score quickly, and so with only minutes remaining, the game was tied. Then the Rockies scored again, this time a long, sharp-angled shot that squirted through my legs. The game had seemed finally lost. But in the last three minutes, Lapointe scored, then Lafleur, and we won 6-5. Alone in the dressing room later .... I just sat there, unable to understand why I felt the way I did. Only slowly did it come to me: I had been irrelevant; I couldn't even lose the game."

In Dryden's career, I found 12 Canadiens games against Colorado. Montreal won 11 of them and tied one. But none of them was by a score of 6-5.

Montreal did not have any 6-5 wins at all in 1978-79 (when the book is set), or in the previous two seasons. In Dryden's entire career, I found only two 6-5 Montreal wins where he was in net.

One was February 12, 1972, against the Kings. The narrative doesn't match. In that game, the Habs led 6-3, and then the Kings scored two late goals.

The other was November 16, 1972. In that game, Dryden was replaced by Michel Plasse after the first period, so that doesn't match either.

So, I extended the search to look for all games where Dryden gave up 5+ goals, but the Canadiens won anyway. There were five such games:

February 12, 1972, 6-5 against LA (described above)
November 22, 1974, 7-6 against Kansas City
February 18, 1976, 7-5 against Toronto
November 21, 1976, 9-5 against Toronto
December 23, 1977, 7-5 against New York Islanders.

None of the games match exactly, but the Kansas City game is the best candidate:

-- It was against the team that eventually became the Rockies;

-- The opposition scored late (14:49 of the third), and the Habs won it later (17:53);

-- It was a one-goal game;

-- Dryden probably didn't play great (6 goals on 21 shots, seven per period);

-- The last five goals alternated by team.

But other things don't match:

-- The Scouts tied it late, not took the lead late;

-- The Habs scored one goal to win, not two;

-- The goal was scored by Doug Risebrough, not Lapointe or Lafleur -- in fact, neither Lapointe nor Lafleur scored at all that game;

-- The early goals didn't alternate (The goal sequence was kMMMMkkkMkMkM);

-- The game was in Kansas City, not Montreal;

-- The game happened several years previously, rather than months.

I checked the Globe and Mail recap for that game ... it was just a couple of paragraphs long, and didn't mention Dryden at all, or how the Kansas City goals went in. I don't have online access to any Montreal newspapers to get a more detailed game story.

-----

2. Number 2 in my blog post listed a game against Toronto. Dryden writes that the Leafs tie the game early, get confident that they can keep up with the Canadiens, and begin to take over the play. But Mark Napier and Pat Hughes score two quick goals for the Habs. The Canadiens score two more, and then the Leafs get two late. The next day, the players wonder why coach Scotty Bowman didn't give them hell for allowing those two late goals.

It all adds up to 6-4. There was no 6-4 win in Toronto in 1978-79.

So, I looked for other games that might match.

There were only two games during Dryden's career where Napier and Hughes both scored.

One was November 15, 1978, a 6-1 win over Colorado. That doesn't match.

But the game of January, 17, 1979 is probably it. It matches, though not exactly:

-- It's in 1978-79, the season Dryden was writing about.

-- It was against Los Angeles, not Toronto, and home, not road.

-- Although the Kings scored to make it 2-1 at 8:43 of the second period, they never tied it up.

-- After the Kings' goal to make it 2-1, Napier and Hughes scored to make it 4-1.

-- After that, the Habs scored three more goals (not two): Houle, then Napier and Hughes again.

-- After that, the Kings got their two late goals.

-- The final was 7-3, not 6-4.

But, I think, pretty close anyway.

Do NHL referees call "make up" penalties? Part III

2012-01-06T12:38:00.004-05:00

The last two posts talked about how NHL referees are more likely to "even-up" their calls, issuing the next penalty to the opposing team 60% of the time.

This post, I'll show a regression I ran to quantify the effect a bit better. If you're not interested, just skip the technical parts (smaller font). If you're *really* not interested, you can probably just skip this entire post, since the results are pretty much the same as shown in the previous posts.

---

Technical notes 1:

Even though we're interested in whether the referee called the next penalty to the "other" team, I set up the regression to predict whether the referee called the next penalty to the *home* team. That just makes everything easier to interpret, but, as I'll describe, it still lets us estimate the "even-up" effect.

In the study, I ignored all misconduct penalties, all first penalties of the game, and all penalties where the other team had a player called at the same time. (I treated those penalties as if they didn't exist, so they didn't interrupt "consecutiveness" of the two surrounding penalties.)

Non-dummy variables I used: Time gone in game. Time since last penalty.

Dummy variables I used: PP goal on last penalty. SH goal on last penalty. Home team lead, from -3 to +3, except 0 (that is, six dummy variables), where anything more or less than 3 goals was coded as 3. All eight of the previous variables interacted with "whether the last penalty was to the home team." And, of course, the dummy for "whether the last penalty was to the home team" itself.

The regression shows the home team percentage diminishes during the game, by about 1 percentage point per period. In all the numbers in this post, I just used the beginning of the game. If you want the middle of the game, subtract about 1.5 percent from each "home team" percentage (or add 1.5 percent to each "visiting team" percentage) if you want to adjust to the middle of the second period.

Also, the regression says you have to subtract about 1 percentage point for every 20 minutes since the last penalty. I didn't bother for this post. If you assume penalties are usually around 5 minutes apart, feel free to subtract 0.25 percentage points from each of the "home team" percentages.

Those two time adjustments won't affect the "even-up" numbers, just the raw percentages of home team penalties.

-----

OK, first, let me show you the percentage of penalties taken by the home team, by game score. Clearly, teams are more likely to take penalties when they're ahead in the game.

After the visiting team took the last penalty, the home team took:

46.3% when down by 3+
49.2% when down by 2
53.1% when down by 1
61.2% when tied
64.3% when up by 1
67.9% when up by 2
66.8% when up by 3+

And after the home team took the last penalty, the home team took:

34.5% when down by 3+
32.8% when down by 2
32.3% when down by 1
35.0% when tied
42.6% when up by 1
47.6% when up by 2
52.9% when up by 3+

Obviously, you can do this for visiting teams just by subtracting all the percentages from 100.

-------

Now, we can calculate the "even-up" effect as the difference between the lines of the two tables. When the score was tied, the home team took:

61.2 percent after visiting team penalty
35.0 percent after home team penalty
-------------------------------------------------
26.2 percent difference

You can convert to visiting team just by subtracting the first two numbers from 100%. The difference has to come out the same. I'll do that anyway. When the score was tied, the visiting team took:

38.8 percent after visiting team penalty
65.0 percent after home team penalty
-----------------------------------------------------
26.2 percent difference

It turns out the "even-up" difference is highest for tie games. Here's the full breakdown:

11.8 percent difference down by 3+
16.4 percent difference down by 2
20.7 percent difference down by 1
26.2 percent difference tied
21.7 percent difference up by 1
20.3 percent difference up by 2
13.9 percent difference up by 3+

------

Tango suggested there might be an extra "compassion" effect when the team scores a PP or SH goal. He seems to have been right. The effect is small, relative to the overall effect, but still enough to affect the games:

-- If the home team scored a PPG on the previous penalty, add 1.2 percentage points to the above differences.

-- if the visiting team scored a PPG on the previous penalty, subtract 3.2 percentage points to the above differences.

-- if the home team scored a SHG on the previous penalty, add 2.8 percentage points from the above differences.

-- if the visiting team scored a SHG on the previous penalty, subtract 0.7 percentage points from the above differences.

The PPG numbers are statistically significant. The SHG numbers aren't, but they go in the right direction and are about the right magnitude, so I think it's reasonable to consider them as decent estimates.

------

So, as I promised in the second paragraph: the results of the regression seem to match what we found in the previous posts.

------

Technical notes 2:

For full disclosure, here are the coefficients for all the variables in the regression. I'll present them in "here's how to calculate the percentage of home-team penalties" format. (If you prefer a table, the full computer output is here (pdf). You'll be able to tell what the variables represent by matching the coefficients to what's below.)

Start with 0.6119 (constant).

Add -0.2619 if the home team took the last penalty.

Add -8.15E-06 for each second that's passed in the game. (About -.01 per period.)
Add -7.39E-06 for each second that's passed since the last penalty (not significant, p=.104, but magnitude is reasonable and has the right sign).

Add -0.1483 if the home team is down by 3 or more goals.
Add +0.1436 if the home team is down by 3+ and also took the last penalty.

Add -0.1196 if the home team is down by exactly 2 goals.
Add +0.0983 if the home team is down 2 and also took the last penalty.

Add -0.0807 if the home team is down by 1 goal.
Add +0.0545 if the home team is down 1 and also took the last penalty.

Add +0.0309 if the home team is up by 1 goal.
Add +0.0450 if the home team is up 1 and also took the last penalty.

Add +0.0668 if the home team is up by 2 goals.
Add +0.0591 if the home team is up 2 and also took the last penalty.

Add +0.0559 if the home team is up by 3 or more goals.
Add +0.1228 if the home team is up 3+ and also took the last penalty.

Add +0.0115 if a PP goal was scored on the last penalty
Add -0.0438 if a PP goal was scored on the last penalty and that penalty was to the home team.

Add -0.0072 if a SH goal was scored on the last penalty
Add +0.0350 if a SH goal was scored on the last penalty and that penalty was to the home team.

-----

Do NHL referees call "make up" penalties? Part II

2012-01-03T10:15:00.010-05:00

Last post, I found that referees are likely to "even up" their penalty calls: they're around 50% more likely to give the other team the next power play than to give the same team two power plays in a row.

I wasn't not convinced this is because of referee bias, or what Tango calls the "compassionate referee."

Tango suggested this experiment: check to see if a power play goal was scored on the first penalty. If the referee is indeed "compassionate" towards the other team, he should be more compassionate if the penalty actually cost them a goal, less so if there was no goal, and even less so if the penalized team *benefited* from the penalty by scoring shorthanded.

So I checked. I looked at all cases where there was a power play goal (PPG) on a first penalty, and then no more scoring until the next penalty was called. Indeed, that does appear to make the ref more compassionate.

After a penalty resulting a PPG, the next penalty was of the "even-up" variety 65.9% of the time. That's higher than the overall rate of 59.7%. Repeating that in a better font:

65.9% after a PPG
59.7% overall rate

And, the same effect appears for shorthanded goals (SHG):

52.5% after an SHG
59.7% overall rate

It's a large effect, and exactly in the direction Tango predicted.

-----

But wait! It might not be referee bias at all. Because, it turns out that teams with a lead take significantly more penalties than teams who are behind. For instance, when a penalty is called while you have a two goal lead, there's a 55.2% chance the penalty goes against you (and so a 44.8% chance the penalty goes against the other team). Full chart:

55.2% of penalties to team leading by 1
58.2% of penalties to team leading by 2
59.0% of penalties to team leading by 3
59.4% of penalties to team leading by 4
59.7% of penalties to team leading by 5

So, the score effect could explain what we're seeing. After a power play goal, the team has a bigger lead (or smaller deficit) than before. That would make it likely to take more penalties in future, even if the referee wasn't compassionate at all.

(Of course, the score effect might itself be due to referee "compassion," but that's a whole other argument.)

Specifically: a power play goal makes the team 6 percentage points more likely to take the next penalty. But scoring ANY tiebreaking goal in the first period makes a team 5 percentage points likely to take the next penalty. So how can we be sure there's a separate power-play effect, or how big it is?

-----

What might also complicate things is there's a "time of game" effect:

42,721 PPs came in the first period.
38.060 PPs in the second period.
26,705 PPs came in the third period.

There are fewer penalties in the third period than in the first. Is that a separate period effect? It might be.

Here's the score effect chart, again, but this time only for first-period penalties. The effect is more extreme than for the entire game:

55.6% of penalties to team leading by 1
60.1% of penalties to team leading by 2
61.0% of penalties to team leading by 3
66.5% of penalties to team leading by 4
58.7% of penalties to team leading by 5 (only 46 datapoints)

-----

It almost looks like we need a regression to sort all this out. But, wait! One more try before we turn to the dark side. Let's engineer a comparison where score and period won't screw things up.

I took every situation where:

1. It was the first period.

2. The game was tied at the time of the first penalty, and exactly one additional goal was scored before the second penalty.

3. The one extra goal was scored by the team that had the power play on the first penalty.

Then, I divided those situations into two groups.

The "Highest compassion" group is where the team scored the goal *on the power play*, presumably making the referee feel extra bad that he caused the goal. The "Typical compassion," is where the team scored the goal *after* the power play, and the referee's call wasn't the cause.

What percentage of the second penalties went to the other team?

Highest compassion: 71.6% (2163 datapoints)
Average compassion: 69.7% (1051 datapoints).

There's a small effect there, in the expected direction, of 1.9 percentage points. (That's less than 1 SD, so not statistically signficant.)

Here's the same result, but the other way, where it's the originally-penalized team that scored before the next penalty. When that goal was scored shorthanded, we can call that "Lowest compassion". When it wasn't, it's again "Average compassion."

Again, what percentage of the time did the second penalty even things out?

Lowest compassion: 62.5% (253 datapoints)
Average compassion: 58.2% (1006 datapoints).

This time the effect goes the "wrong" way, but there's too little data to draw any conclusions.

Doing the same thing for the second period instead of the first, we find a larger difference, but still not statistically significant (1.4 SD):

Highest compassion: 70.4% (568 datapoints)
Average compassion: 65.7% (271 datapoints).

And the shorthanded case, which really has too small a sample to take seriously:

Lowest compassion: 51.8 (83 datapoints)
Average compassion: 48.9% (268 datapoints).

-----

So, in summary: yes, there appears to be weak evidence for a small "compassion effect."

In the previous post, I considered three hypotheses:

1. Referee bias
2. Penalized teams play more carefully after the penalty
3. Power play teams play more aggressively after the penalty

Here's a fourth one, a variation of one suggested by commenter Wexler in the previous post:

4. Referees like to let the players play, and dislike calling penalties. But, sometimes they have to assert themselves to make sure the game doesn't get out of hand. Sometimes they're a bit too late, and they have to call a penalty on something that wasn't a penalty two minutes ago. This sends a message to the players, "OK, enough."

That might be necessary, but is obviously unfair to the penalized team. And, so, the referees know they have to call a "make up" penalty on those particular calls. Both teams understand what's happening, and won't object to either that call or the subsequent call.

I don't know if #4 is plausible or not ... but one of my co-workers is a soccer referee, and it's consistent with what he says about having to keep the game under control before it's too late.

As usual, I await comments from readers who know more about this stuff than I do.

------

UPDATE: Part 3 is here.

Do NHL referees call "make up" penalties?

2011-12-31T13:06:00.009-05:00

Among NHL fans, there's a perception that referees like to call "make up" penalties. If a ref has just called a minor penalty on one team, it's very likely that the next penalty will go to the other team.

I was skeptical, until I downloaded a bunch of data from The Hockey Summary Project ... they're like Retrosheet for hockey. (Their website is here, and if you want data downloads, you can join their group by going here.)

I looked at all penalties from 1953-54 to 1984-85 (for which the HSP data is almost complete). I eliminated all cases where there both teams got penalties at the same time. Then, I checked what was left, to see if the team that got the current penalty was less likely to get the next one.

Absolutely, very much so. There's a 60% chance the next penalty will go to the other team -- 59.7%, to be more exact. (But, since I'm not sure that database is complete, and I forgot to remove misconducts, and I didn't consider situations where both teams got a penalty but one team got an extra one, I'm happier to drop the decimal and just go with 60%.)

The effect is reasonably consistent over time, although it was a little stronger back in the six-team era. Here's a too-long chart.

1953-54: 62.7 888/1416
1954-55: 61.3 857/1397
1955-56: 60.6 912/1506
1956-57: 61.2 833/1360
1957-58: 58.8 793/1348
1958-59: 61.4 801/1305
1959-60: 62.3 723/1160
1960-61: 61.7 740/1199
1961-62: 62.0 821/1324
1962-63: 62.0 797/1286
1963-64: 61.8 826/1337
1964-65: 61.7 841/1362
1965-66: 58.6 820/1399
1966-67: 58.6 710/1212
1967-68: 60.0 1515/2527
1968-69: 59.5 1666/2800
1969-70: 60.5 1793/2962
1970-71: 60.4 1944/3220
1971-72: 57.8 1918/3317
1972-73: 60.6 2200/3633
1973-74: 58.8 2135/3628
1974-75: 56.7 2873/5069
1975-76: 56.3 2890/5130
1976-77: 57.2 2371/4144
1977-78: 59.1 2316/3916
1978-79: 59.5 2337/3925
1979-80: 59.3 3021/5091
1980-81: 60.1 3780/6293
1981-82: 60.7 3613/5957
1982-83: 60.3 3479/5774
1983-84: 60.2 3788/6296
1984-85: 60.6 3542/5847
-------------------------
Overall: 59.7 39597/58543

Even though the effect is real, we can't say for sure that it's referee bias. It could just be that, after a penalty, the penalized team plays more cautiously, trying to avoid a second penalty. Or, it could be that the just had the power play decides to play more aggressively.

(As an aside: why did penalties drop so much between 1975-76 and 1976-77? At first I thought it might be bad data, but then I checked power-play opportunities on Hockey Reference, and it checked out.)

Here's what I think is some relevant evidence. I broke down the stats by referee (minimum 300 datapoints). The database only has the referee named for about a quarter of the total games (mostly older ones), but I figure it's probably good enough to at least look at.

The first column is the main number, the percentage of penalties called against the team who drew the last one.

Pctg Z-sc Size Ref
---- ---- ---- ----------------------------
59.3 00.0 0509 Andy Van Hellemond
60.4 +0.4 0846 Art Skov
59.3 -1.2 1128 Ashley
60.0 00.0 0460 Bill Friday
57.0 -1.1 0537 Bob Myers
58.9 -0.3 0878 Bruce Hood
57.1 -0.9 0580 Bryan Lewis
59.3 -1.6 1453 Buffey
64.6 +1.6 0933 Chadwick
59.4 -0.1 0567 Dave Newell
60.4 -0.3 0356 Farelli
60.1 -0.3 0511 Friday
59.9 -0.1 0696 John Ashley
61.3 +0.8 0359 Lloyd Gilmour
61.3 -0.2 0789 Macarthur
56.7 -1.4 0319 Mehlenbecher
63.0 +0.5 0327 Olinski
60.4 -0.8 1906 Powers
55.2 -2.2 0698 Ron Wicks
63.7 +1.7 1095 Skov
57.7 -3.1 2197 Storey
63.1 +2.7 4717 Udvari
59.9 +0.4 0709 Wally Harris

The least "biased" referee is 55%, and the most "biased" is 64%. If you think it's only referee bias that keeps the numbers from being 50%, you'd have to think that EVERY referee is biased almost exactly the same way. It's hard for me to accept that none of the referees noticed the bias and saw fit to try to eliminate it.

The second column of the table is the Z-score, the number of standard deviations the referee is from expected (which is normalized to the seasons he officiated). Normally, you concentrate on those with at least plus or minus 2 SD. That gives you Red Storey and Ron Wicks (less biased than most) and Frank Udvari (more biased than most).

The standard deviation of the Z-scores was 1.29. If every referee were the same, and differences were only random, it would be 1.00. This suggests that there are real differences between referees. Specifically, the SD of referee tendencies (or "talent", you might say) is 0.8 (since 1 squared plus 0.8 squared equals 1.29 squared).

In English, you can perhaps interpret that as saying that the differences in the table are about half real and half random, with a little more random than real (since 1.00 is a little higher than 0.8).

The observed range is 55 to 64. Regressing to the mean, the actual range of referee tendencies is probably 57 to 62, or something like that.

So if you think it's referee bias, you have to explain why all the referees seem to be biased within such a tight range, especially, when, presumably, they are all working hard to be as unbiased as possible.

--------

Here's another interesting breakdown, by time since the previous penalty:

0:01 to 1:00: 69.1 (7000)
1:01 to 2:00: 64.7 (9444)
2:01 to 3:00: 68.5 (11778)
3:01 to 4:00: 64.2 (10574)
4:01 to 5:00: 61.2 (8831)
5:01 to 6:00: 59.7 (7470)
6:01 to 7:00: 58.9 (6328)
7:01 to 8:00: 58.3 (5333)
8:01 to 9:00: 56.6 (4399)
9:01 to 10:00: 55.5 (3719)
10:01 to 11:00: 56.7 (3000)
11:01 to 12:00: 55.5 (2591)
12:01 to 13:00: 53.8 (2193)
13:01 to 14:00: 55.2 (1837)
14:01 to 15:00: 53.3 (1565)
15:01 to 16:00: 53.1 (1376)
16:01 to 17:00: 53.1 (1135)
17:01 to 18:00: 52.4 (1019)
18:01 to 19:00: 51.9 (807)
19:01 to 20:00: 53.6 (757)
20:01 to 99:99: 51.8 (3883)

The longer the interval since the previous penalty, the less likely the next penalty will go to the other team. That's consistent with many theories. The "referees are biased" theory would say that referees "forget" to even things up as the game goes on. The "other team wants revenge and plays aggressively" theory would say that if they don't get revenge early, they don't need it as much later. And the "penalized team takes fewer chances" theory would say that as time goes on, the players "forget" that they have to be more careful.

So, the data doesn't help us choose, but it's interesting nonetheless.

By the way, the 1:01 to 2:00 group is an exception to the pattern, but that's probably due to power plays, since the first penalty is probably still in effect. Actually, I'd have expected that part to go the other way, with the first two minutes being *more* than 50 percent, on the logic that the shorthanded team playing in the defensive zone is more likely to be forced to take a penalty. But, that doesn't happen.

And here's an interesting breakdown of the first half of the first group:

81.9% within 5 seconds
78.1% between 6 and 10 seconds
76.0% between 11 and 15 seconds
73.8% between 16 and 20 seconds
69.3% between 21 and 25 seconds
67.0% between 26 and 30 seconds.

-----

Finally, one more question: after one team gets, say, four straight penalties, what happens then? Is there an even stronger bias for the other team to take the next penalty?

Yup.

57.1 after exactly 1 in a row (64858 datapoints)
64.0 after exactly 2 in a row (23850)
66.6 after exactly 3 in a row (7042)
66.0 after exactly 4 in a row (1781)
63.8 after exactly 5 in a row (442)
60.6 after exactly 6 in a row (127)
67.5 after 7 or more in a row (40)

------

So: what's going on? Any ideas?

UPDATE: Part 2 is here. Part 3 is here.

The best goalies should play for the worst teams

2011-12-28T16:12:00.003-05:00

Last week, I described a way to look for "bad team" goalies as described by Ken Dryden. I don't think that method is going to work ... the sample sizes are too small.

But, while doing the math (which I'll spare you), it occurred to me that there IS a class of goalies that could be considered "bad team" goalies, in the sense that they're more valuable to a good team than a bad team. That class of goalies is simply ... the best goalies.

The worse the team defense, the more shots the other team gets. So the great goalie will wind up saving a lot more goals for a bad defense than a good defense. It's like how a policeman is more productive in a bad neighborhood.

I guess this isn't a new realization ... people have said that Ken Dryden was wasted, a bit, in the Montreal goal for so many years ... there was very little for him to do. He would have had more value to a team with a worse defense -- at least in terms of goals saved.

So, if teams are rational, you should see the best goalies playing behind the worst defenses.

Or maybe not. I'm surprised at how tiny the effect is. Eyeballing last year's NHL stats, it looks like bad teams gave up maybe 125 more shots than average.

The best goalie in the league might be 2 SD above average, or .008. Multiply 125 by .008, and you get ... exactly one goal. So, even a great goalie is worth only one more goal to a bad team than to an average team.

That assumes that all shots are equal. Suppose bad teams give up harder shots, and good teams give up easier shots. Maybe that doubles the effect. In that case, the advantage becomes two goals. So moving from the best team to the worst is worth four goals -- from two goals worse than normal, to two goals better than normal.

Hmmm ... maybe not as tiny as I thought. To get four goals of goalie improvement is the equivalent of 3/4 of a standard deviation in goalie talent. For a team, saving four goals should get you, what, maybe a couple of points in the standings?

------

I'm sure this is old hat to hockey sabermetricians, but this is the first time it seriously occurred to me that the same player can be more valuable, in terms of influencing the score, with a bad team than a good team.

You've also got the punter in football ... the worse the offense, the more fourth downs, so the more important punting is overall. And, maybe, the safety: he's the last line of defense, so he gets more chances when his teammates fail to make the tackle before him.

In baseball, good fielders are more valuable on bad teams, since bad pitchers allow more balls in play. Also, a bad team will have a lot more men on base than normal, which means more double-play opportunities. Also, a strikeout pitcher is more valuable on a team that doesn't field well.

You might also argue for the NHL enforcer, if his job is to start fights when his team is behind, and you also accept the premise that the goonery actually helps the team come back.

Which is the strongest example, the one where the player adds the most value moving to the bad team? I'd guess the NHL goalie, but, really, I have no idea.

Will taxing the rich improve democracy?

2011-12-27T11:09:00.013-05:00

Many people believe that income inequality in our society is too high. I generally don't agree (some of my reasons are here), but I'm open to arguments. However ... not the particular argument that Ian Ayres and Aaron Edlin made in the New York Times last week (and later added to at the Freakonomics site), that income inequality undermines democracy:

The progressive reformer and eminent jurist Louis D. Brandeis once said, “We may have democracy, or we may have wealth concentrated in the hands of a few, but we cannot have both."

Brandeis understood that at some point the concentration of economic power could undermine the democratic requisite of dispersed political power. This concern looms large in today’s America, where billionaires are allowed to spend unlimited amounts of money on their own campaigns or expressly advocating the election of others.

What we call the Brandeis Ratio — the ratio of the average income of the nation’s richest 1 percent to the median household income — [now stands at 36]. We believe that we have reached the Brandeis tipping point. It would be bad for our democracy if 1-percenters started making 40 or 50 times as much as the median American.

Enough is enough. Congress should reform our tax law to put the brakes on further inequality. Specifically, we propose an automatic extra tax ["Brandeis Tax"] on the income of the top 1 percent of earners -- a tax that would limit the after-tax incomes of the club to 36 times the median household income.

So, their idea is: some people have so much money more than the rest of us that they can disproportionately affect the outcome of elections. Therefore, we should tax them to make the income distribution more equal. That way, the rich will have less money, which means they'll spend less influencing politicians, and democracy will be stronger.

There are so many reasons to disagree with this that I won't list them all. But, the most obvious one: are the authors really saying, with a straight face, that a tax on the rich will make them less politically active? Would that work with any other group? "Hey, black people are starting to march on Washington. Let's put a special surtax on black people. That'll quiet them down!"

And that's not even the biggest problem. The biggest problem is that Ayres and Edlin don't own a mirror.

Think of the most important ways that the USA and Canada are different, in terms of government policy, than they were a couple of generations ago. If you were to ask me, the biggest changes are things like: The elimination of racial segregation. Women's Rights. Gay Rights. The Canadian Constitution and Charter of Rights. A more peaceful world. A better welfare system. Somewhat lower crime. Socialized medicine (in Canada).

Add your own. Then, ask yourself, how many of them have much to do with rich people giving money to politicians?

Take, for instance, racial segregation. How did that change? Did some rich black guy come along, slip a politician a couple of million dollars, and then, wham, suddenly everyone gets to eat at the same lunch counter?

Of course not. Racial progress happened by a change in the attitudes of Americans, not by the actions of politicians (who acted late, only in response to public demand.) What happened is that Americans fueled the process. They read the newspaper, and debated, and protested, and wondered, and marched, and argued, and pondered, and chatted at the water cooler, and gave speeches, and rode buses, and watched TV.

And, slowly, day by day, week by week, people's views changed.

It had nothing to do with where wealth was concentrated, and nothing to do with the richest 1% of Americans donating money to the right politicians. The "power" WAS dispersed; it was dispersed among millions and millions of citizen voters. The wealthiest black person in the country couldn't possibly have used political contributions to speed up the process much before its time.

Of course, some people had more influence than others, like Martin Luther King. And, the media: newscasters, and columnists, and reporters; TV, and radio, and newspapers. Someone writing for the New York Times, for instance, would get read by millions of people. One New York Times is the equivalent of thousands of water coolers.

That means that if you want to argue that inequality is undermining democracy, you shouldn't be thinking about money. You should be thinking about public discourse.

Let's suppose that over the last year, the Times has had five or six opinion pieces per day. That means that, at most, around 2,000 Americans got to have their voices heard in the op-ed pages of the New York Times. Many writers, of course, appeared more than once. For the sake of argument, let's say there were 1,500 different writers. The US population is 300 million, which gives us this shocking measure of inequality of influence:

The top 0.0005% of US writers wrote 100% of the New York Times op-eds.

Compare that to money: by my estimation, the top 0.0005% of US households earned less than 2% of the overall income.

By this measure, that means the concentration of op-eds is FIFTY TIMES AS HIGH as the concentration of income. Fifty times.

But that's not really fair, since the New York Times isn't the only place you can broadcast your influence. Let's suppose the top 100 venues -- newspapers, TV, or blogs -- have 100,000 different writers, and comprise 75% of the total political influence in the US. Compare that to the top 100,000 households in income:

0.03% of the population has 8% of the income.
0.03% of the population has 75% of the influence.

So, if democracy is compromised by the unequal distribution of income, how can you say it's not compromised by an unequal distribution of public access to opinions and influence -- especially when the latter inequality is EIGHT TIMES AS HIGH?

The fact is that Ian Ayres, with at least three other op-ed pieces in the New York Times in the past ten years -- 50,000 times as many as the average American -- has much, much more influence on public policy than your typical rich guy. And Paul Krugman, the Times economics columnist, writes around 100 columns a year -- fifteen million times as many as the average American.

"Enough is enough," indeed.

I propose a "Krugman Tax." Every year, we'll compute the "Krugman Ratio," the number of words the top 100,000 writers published, divided by the number of words the average American published. If it's more than, say, 40, those writers will be taxed just enough words to bring the ratio back down to 40 in future. That will ensure that everyone, not just the ultra-published like Ian Ayres and Paul Krugman, can influence American political decisions.

We'll do it for democracy.

Are there "good team" goalies and "bad team" goalies?

2011-12-21T15:36:00.006-05:00

In "The Game," Ken Dryden argues that some players are not psychologically suited to playing on good teams:

Because the demands of a goalie are mostly mental, it means that for a goalie the biggest enemy is himself. The fear of failing, the fear of being embarrassed ... The successful goalie understands these neuroses, accepts them, and puts them under control. The unsuccessful goalie is distracted by them, his mind in knots, his body quickly following.

It is why [Rogie] Vachon was superb in Los Angeles and as a high-priced free-agent messiah, poor in Detroit. It is why Dan Bouchard ... lurches annoyingly in and out of mediocrity. It is why there are good "good team" goalies and good "bad team" goalies -- Gary Smith, Doug Favell, Denis Herron. The latter are spectacular, capable of making near-impossible saves that few others can make. They are essential for bad teams, winning them games they shouldn't win, but they are goalies who need a second chance, who need the cushion of an occasional bad goal, knowing that they can seem to earn it back later with several inspired saves. On a good team, a goalie has few near-impossible saves to make, but the rest he must make, and playing in close and critical games as he does, he gets no second chance.

A good "bad team" goalie, numbed by the volume of goals he cannot prevent, can focus on brilliant saves and brilliant games, the only things that make a difference to a poor team. A good "good team" goalie cannot. Allowing few enough goals that he feels every one, he is driven instead by something else -- the penetrating hatred of letting in a goal.

Dryden seems to be saying at least three things here:

1. Some goalies, like Rogie Vachon, can't handle pressure.
2. Some goalies are better on bad teams than on good teams.
3. Those two groups are the same goalies.

I'm very skeptical about #1, especially with regards to Rogie Vachon. Yes, Vachon had a serious decline after leaving the Kings -- with Detroit, he was worse by more than a goal a game (3.90 to 2.86). But, was it really Vachon's neuroses? After all, he was 33 years old that year. Dryden may know Vachon pretty well -- they were together on the Canadiens for a few months in 1971 -- but is that enough for him to conclude that Vachon's problem is that he choked under pressure?

I'll skip over #3, also, and concentrate on #2, the part about "good team" goalies and "bad team" goalies. What Dryden seems to be saying, as an empirical hypothesis, is something like this:

There are some goalies who make brilliant saves that few others can, but also give up more weak goals. Those goalies are more valuable to bad teams, because bad teams give up more scoring chances where brilliant saves are required. It wouldn't make sense for a good team to pick up a goalie like that, because they'd get only the weak goals, but not the brilliant saves.

That's actually a pretty interesting theory! And it seems plausible. After all, what a team should care about is how many goals a guy allows, not how he looks doing it. A goalie with a 2.50 GAA is more valuable than a goalie with a 2.75 GAA, even if the first guy lets in more bad goals than the second guy.

But, is there any evidence for it?

Not in the book. Dryden gives us only those three examples of "bad team" goalies. Unfortunately, they played on bad teams for most of their careers.

Still, we have a few datapoints.

-- Gary Smith left Oakland (bad) to play two seasons for the Black Hawks (good) as Tony Esposito's backup. The first year, he was very good; the second year, he was mediocre.

-- Denis Herron moved from (bad) Pittsburgh to (good) Montreal (where he replaced Ken Dryden). Like Smith, he was great the first year, but not so great the second year.

-- Doug Favell was nothing special in his last season with Toronto (an average team). Then, he went to a below-average Colorado Rockies team, where it seems like he was pretty good. So, maybe that's a plus.

But, overall ... no real evidence either way, really. And Dryden doesn't give any examples of "good team" goalies, so there's nothing to check there.

But ... maybe here's something we can do.

Goalies play some of their games against good teams, and some against bad teams. If Dryden is correct, that Smith, Herron and Favell give up more bad goals but also make more spectacular saves, they should do better than expected against good teams, and worse than expected against bad teams. That's because they'll give up roughly the same amount of bad goals each way, but they'll make more brilliant saves against the good teams.

Does that make sense? Maybe we can find a way to check that.

Here's how that might work. In 1977-78, the Penguins, with Denis Herron as their regular goalie, gave up 321 goals, or 4.01 per game. That season, the best five teams (alphabetically) were the Bruins, Canadiens, Flyers, Islanders, and Sabres. The worst were the Barons, Blues, Canucks, Capitals, and North Stars.

From the game log, I manually calculated that against the five best teams, the Penguins gave up 5.25 goals per game. Against the five worst teams, they gave up 3.14.

For that to be evidence that the Penguins have "bad team" goalies, you'd have to show that 5.25 goals against good teams is actually better than expected for a team that gives up 4.01, and that 3.14 against bad teams is actually worse than expected for a team that gives up 4.01.

How would you do that? Well, one thing you could do is find a matching team, one that also gave up 4.01 goals per game (or close to it), but had a random goalie. If that team gave up 6.00 goals against the good teams, but 2.50 against the bad teams, that would be confirmatory evidence.

The match wouldn't be perfect, because it might have to be from another season, and the "best" and "worst" groups might not be comparable. Still, even with just those three goalies, you'd have about 33 seasons to compare (if you require a minimum 30 game season). If you found the two most comparable instead of one, that would be 66 comparisons.

That's better than nothing. But there'd still be a lot of noise. To make it workable, you'd have to limit your sample to games those goalies started (in 1977-78, Herron himself gave up only 3.57 goals per game, compared to 3.96 for the team after subtracting empty net goals). There'd be even less noise if you used save percentage instead of goals against.

The process would be a bit like searching for clutch hitting, only with a lot less data. And it would be a lot of work ... but, there's an organization, the Hockey Summary Project, that collects NHL game summaries -- like a hockey Retrosheet -- and I've asked them for access to their database. I'm hoping they have shots on goal. (Also, those summaries might help us trying to trace the games that Dryden talked about in his book, the ones that didn't match the game logs.)

Before I go any further, does this make sense as a way to test Dryden's hypothesis? Can you think of any others that might be easier?

Ken Dryden's "The Game"

2011-12-19T09:49:00.014-05:00

Ken Dryden's book, "The Game," is considered one of the greatest ever in sports. Many times, I've heard it called "the best hockey book ever written," and that's the quote (unattributed) on the cover of my 1999 edition. The Canadian literary crowd loves it.

So, last week, I thought I'd read it. I was disappointed.

"The Game" takes place towards the end of the 1978-79 NHL season, Dryden's last before retirement. It was actually written later, based on his notes at the time, and published in 1983. It takes the form of a ten-day diary, although most of each day is taken up by Dryden's reminiscences and analyses, rather than the actual events of the particular day.

I'd argue that it's only tangentially a book about hockey. It's really a book about Ken Dryden. What he does, how he feels, what goes on in his dressing room, what he thinks of his teammates, and so forth. If you actually were hoping to learn something about hockey and how it works, there won't be much for you here ... except that you'll hear some stories about the personalities of teammates and coaches.

It doesn't talk much at all about strategy, or playmaking, or statistics, or how to win games. It deals a little more with personalities, and stories. But, mostly, it deals with Ken Dryden's feelings.

That may not sound too interesting ... except that Ken Dryden is very, very articulate. He can take the most common observation, and write paragraphs of poetry about it. Here he is talking about travelling to the Forum:

"I drive down side streets narrowed by drifts and snow-shrouded cars. Traffic is light today, and the few cars on the road move easily, unconcerned by the conditions. After the awkward caution of a winter's first snowfall, for Montreal drivers, like riding a bike, it all comes back, and slippery streets are driven as if bare and dry. I park several blocks away from the Forum and walk. The wind, gusting up Atwater Street, is bitter and cold, and, hunching over, I try to cover up, but can't. I start to jog, then run, faster as the wind bites harder. At de Maisonneuve, the light turns red but I continue across."

Well, OK, great, that's certainly a vivid picture of a windy Montreal winter's day. But the tone is a bit dramatic. It's an impressive feat of writing, and there's nothing wrong with being Shakesperean at times ... but Dryden does it *everywhere*, from the first page to the last. It started grating on me.

What would mitigate the tone, a bit, is if Dryden would talk more concretely about hockey. Sure, there are a few pages on Scotty Bowman's coaching style, and on Larry Robinson's history, and little profiles of some of the other players pop up occasionally. Finally, close to the end of the book, there's a bit of meat -- about 20 pages on the history of hockey, and how the game has suffered, and how rule changes can fix it. It's really good stuff, especially for 30 years ago, when that kind of analysis was rare. Moreover, with something more concrete to explain, Dryden allows some air to come out of the lofty prose, and it reads a lot better.

Still, even accounting for those 20 pages, the book is mostly Ken Dryden, sociologist and psychologist, observing his hockey team. It becomes a little weird because Dryden writes like he's not even there, like he's a psychiatrist floating above the dressing room taking notes.

I guess that's his thing. Ken Dryden is a lawyer; he famously took a year off from the NHL in 1973-74 to finish his law degree. He's intelligent and articulate, and it's almost like he figures there's no everyday scene that can't be made better if you try to analyze it dramatically:

"... Half-naked players move hurriedly about, laughing, shouting for tape (black or white, thick or thin), cotton, skate laces, gum, ammonia "sniffers," Q-Tips, toilet paper, and for trainers to get them faster than they can. It is the kid of unremitting noise that no one hears and everyone feels. But there is another level dialogue we can all hear. It is loud, invigorating, paced to the mood of the room, the product of wound-up bodies with wound-up minds. It's one line, a laugh, and get out of the way of the next guy -- "jock humor." It is like a "roast," the kind of intimate, indiscriminate carving that friends do to keep egos under control. Set in motion, it rebounds by word association, thought association, by "off the wall" anything association, just verbal reflex, whatever comes off your tongue, the more outrageous the better. Elections, murders, girl friends, body shapes, body parts, in the great Tonight Show / Saturday Night Live tradition, verbal slapstick dressed as worldly comment ..."

Nothing wrong with all that, but ... for a lot of the book, that's all there is. And, for all the analysis and introspection, you won't even feel like you know much abour Dryden, which is weird, because he writes a lot about himself: how he feels, and when he's more confident in net, and how he reacts to winning and losing, and his thoughts on retirement. But it's all detached, like he's trying to save you the trouble of understanding him yourself by doing the analysis for you.

There are a few occasions when he tries to say something about the game, something that a sports columnist would say, and it's almost a relief -- finally, something you can get your brain around! At one point, for instance, he talks about how he doesn't think the Canadiens are going to win this year (they eventually did, but that season was their last of four consecutive Stanley Cups). Why? Because, he says, he notices the team is complacent, spoiled by its own success, looking for the "big play" to win the game. Players "shoot from long range, safe from the punishment that goes with rebounds, deflections, screens, and goal-mouth tip-ins." Dryden says he himself cares less about winning, "content that goals appear as "good goals.""

I'm not sure I necessarily buy it 100%, but at least there's something concrete there ... you can try to figure out if it's true or false, or at least how you could study the issue. You think, hey, the guy played in the NHL for a few years, finally he's telling us what he learned!

But, that's as meaty as it gets before Dryden starts getting poetic again:

"I have felt it before in other years, but never so often and never with the same feeling, that if we lose, it will be because of us, no one else. It is not fun to feel a team break down, to find weakness where I always found strength; to discover the discipline and desire can go soft and complacent; to discover that we are not so different as we once thought; to realize that winning is the central card in a house of cards, and that without it, or with less of it, motivations that seemed pure and clear go cloudy, and personal qualities once noble and abundant turn on end; to realize that I am a part of that breakdown."

I can see why some people like this stuff, but ... well, I don't. It's just cotton candy, an exquisitely-worded paragraph that melts down to nothing when you try to figure out what it means. It's articulateness disembodied from communication, as if, when you say something beautifully and poetically enough, it doesn't matter that there's no content. Dryden produces this huge fog of articulateness that overwhelms you with feeling and the sense that something important has just been said, but ... there's not much there when you actually look.

I can't resist one more example. Here's Dryden analyzing Guy Lafleur:

"For there is a life there, and in destiny and romance there is no room for life. Painted as they are with broad brush strokes, vivid and lush, they find shape and pattern only with distance. The person who lives them is too close. He feels sweat as well as triumph. He understands what others see, but feels none of it himself."

Huh? Some people eat this stuff up, but I just don't get it.

------

Call me cynical, but I think that these things I think are weaknesses are actually why people like this book. It's about hockey, but not about hockey. It talks about "big issues" that people like to pretend they care about. There are long digressions into Quebec separatism and culture, and personal growth and emotion. It's articulate, it's educated, and it's erudite. It detaches the reader from the world of uneducated jocks, allowing them to identify and affiliate with someone they look up to. For some readers, it lets them signal to the world that *this* is the hockey they like, that they watch every Saturday night and talk about at the water cooler, the hockey that's deep and sociological and high-class.

"The Book" is a lot like a politician giving a speech, saying things so beautiful and eloquent and moving that you don't even notice that he's not saying anything concrete about what he'll do once elected. You wind up voting for the guy, not because of how he'll be able to run the country, but because you feel like you're voting for him as a person, good-looking and well-dressed and articulate.

But, no offense to Dryden ... it could be just me, that his book just isn't what I'm looking for, that the emotional stuff doesn't do anything for me. But ... whether it's me or not me, this is still not a book about hockey. It's a book about Ken Dryden, articulate hockey player. If you didn't know much about hockey before the book, you still won't know much about it after.

Of course, if you're a Montreal Canadiens fan, you'll love it ... the little stories about the team you love will be irresistible. In his chapter on Toronto, there are some of Dryden's little psychological observations on some of the Leafs of my childhood. I'm not sure if I really believe them, but I still ate them up, and I wished there were more.

------

It's kind of an aside, but the reminiscences in the book don't seem to be right. When I started writing this post, I figured I'd try to let you know which ten days Dryden's diary covered. It should have been easy: a ten-day stretch with the first game in Buffalo, and the last game against the Islanders. But the historical record doesn't jibe with Dryden's narrative. It's not just occasionally that the book is off, but almost all the way through.

This may be a little long ... if you don't like this boring "tracers" stuff, you can skip to the next section.

1. At the beginning of the book, Dryden writes "last night was the sixty-second game of my eighth season with the Montreal Canadiens." It was a game in Buffalo, where the Habs won and Dryden played well.

Here's a game log for the 1978-79 Canadiens, one I'll refer to many times in this post. From the log, it seems the 62nd game of the 1978-79 season was a home game against the Leafs, on March 1. Dryden might be referring to the February 18, game, which was a 5-2 win in Buffalo. Not that big a deal.

That game, Dryden writes, came "after a tie in Chicago and a Saturday loss at home to Minnesota." That doesn't work out. The Canadiens played two games in Chicago that year: a 4-1 loss on October 28, and a 5-3 victory on December 20. Neither of their two home games against the Black Hawks was a tie, and neither closely preceded a game in Buffalo. They did tie Chicago the year before, on Feburary 9, 1978, but that was a home game.

They did play Chicago at home on November 25, losing 8-3, making up for it with a 8-1 rout in Buffalo two games later. That's as close as I could find.

As for Minnesota, the Habs played them four times that year, winning three. The loss was a 4-3 game, but not at home, and not on a Saturday. It was on Wednesday, March 14, 1979 -- almost a month *after* the Sabres game that Dryden possibly describes.

2. That Buffalo game was on a Sunday, as it was described as "yesterday" in the "Monday" chapter. The next game occurs on "Wednesday," at Maple Leaf Gardens in Toronto. That could have been the aforementioned 62nd game that year, on March 1, except that it doesn't match. The Habs won it 2-1, but Dryden's description is 6-4.

It can't be that he just misremembers the score, because he describes the game in detail. After tying the game, Dryden writes, the Leafs get confident that they can keep up with the powerhouse Canadiens, and the Leafs begin to take over the play. But Mark Napier and Pat Hughes score two quick goals for the Habs. The Canadiens score two more, and then the Leafs get two late. The next day, the players wonder why coach Scotty Bowman didn't give them hell for allowing those two late goals.

That adds up to 6-4. There was no 6-4 win in Toronto in 1978-79. Also, I couldn't find a 6-4 win in Toronto in either of the two prior seasons.

There was a 6-3 win on February 3. I looked that game up in the February 5 Globe and Mail. It doesn't match Dryden's description: it went 1-0 for the Leafs, then 3-1 Habs, then 3-2, Then 4-2, then 6-2, then 6-3.

3. The next day, Dryden writes, the Habs fly to Boston, where they win 3-2 after coming back in the third period from a 2-1 deficit. But, according to the 1978-79 game logs, Montreal played only two regular season games in Boston, and tied both of them, 1-1 and 3-3.

There *is* a game that fits in the *following* season, February 10, 1980. It matches the 3-2 score, and the Habs coming back from 2-1 in the third period. But, of course, it can't be that game, because Dryden was then retired (Denis Herron was in net). Also, Dryden describes Larry Robinson tying the game early in the period on a power-play goal, and Mario Tremblay potting the winner. But the third period goals were actually scored by Mark Napier and Pierre Larouche -- the first goal coming at 10:03 -- and there were no power plays after the first period.

4. The next game is in Montreal, against Detroit. Scotty Bowman says, "we got 'em back in Detroit next week." The home-and-home timeline pins it down to the game of April 4 (the return game in Detroit would have been April 8). According to the game log, Montreal won the April 4 game by a score of 4-1.

But that can't be it. During the game, Dryden notices the Leafs vs. Flyers on the out of town scoreboard. Those two teams didn't play each other on April 4. But they did on March 3, when Detroit was also visiting Montreal. So now we have two candidates.

Dryden doesn't explicitly tell us the final score, but he tells us Montreal won. The book says it was 1-0 after the first period, 2-1 after the second, and, with five minutes left in the game, Guy Lapointe scored on a shot off Mario Tremblay's right knee to make it 3-1.

Montreal lost on March 3, so it couldn't be that one. On April 4, Montreal won 4-1, which looks promising -- perhaps an empty-net goal that Dryden didn't report.

But, nope, the rest of the details don't match. The April 4 game was 1-1 after one period, and 4-1 after the second. Jacques Lemaire had a hat trick, and Steve Shutt the other goal.

5. The next game, Dryden has Montreal 7, Philadelphia 3. Flyers goalie Bernie Parent was out with an eye injury that game, the book says.

Parent did indeed suffer a career-ending eye injury in 1978-79, which occurred on February 17. But, Montreal's last game against Philadelphia was January 29. It was indeed a 7-3 score -- but Bernie Parent was the starting goalie.

6. Next is a road game against the Islanders. The book has been foreshadowing that Islanders game from the beginning -- the Islanders were challenging Montreal for number 1 in the standings, and appeared to be the Canadiens' main rivals for the Cup that year. There are several mentions of that important Islanders game coming up, including one in the first few pages, which puts it nine or ten days after the Sabres game.

As far as score, all Dryden says is that the Canadiens lost. So it could be either of Montreal's two games on Long Island: February 27, by a 7-3 score, or October 17, by 3-1. The February 27 game is nine games after the original Buffalo game, so that must be it. Still, none of the games in between are the ones Dryden describes in the book.

7. Epilogue: "In the season's final game, we needed a tie against Detroit for first place, and we lost. The Islanders, waiting to be crowned, lost to the Rangers in the playoffs. And we won again."

Finally, this time it works out. Detroit beat Montreal 1-0 in the last game, and the playoffs match Dryden's recollection.

...

So, six out of seven cases don't check out. What happened? Perhaps instead of using an actual ten days out of the season, Dryden pieced together a composite. That might actually make sense. The book has lots to say about Toronto, where Dryden grew up. It has lots to say about Boston, where he went to school and had a major playoff victory in 1972. And it has lots to say about Philadelphia, which Dryden uses as his springboard for what's wrong with hockey.

He'd have liked to have a period that included all three cities. Maybe he just constructed one out of previous games, periods, and goals that he saw.

...

8. Finally, not a score thing, but: on page 71, Dryden describes Maple Leaf Gardens:

"The enormous Sportimer is gone; an even larger, more versatile scoreboard-clock, the kind you might find in any large arena, is in its place."

But: in 1979, Gardens sported the same, iconic clock that it had since 1966. It wasn't changed until 1982 -- after Dryden's retirement, but before his book was published. Or, maybe he's referring to the original Sportimer, the one before 1966, which he would have seen as a child.

Either way, Dryden's career went from 1971 to 1979, so he would have seen the exact same clock his entire career. So what's this all about? Perhaps he saw the new clock in 1982, before the book was published, and thought he also saw it back in 1979.

------

Reading the book made me think of how different Dryden is from Don Cherry.

Cherry, in one five-minute episode of Coach's Corner, will say more of substance than Dryden says in ten pages. But Cherry uses uneducated language, dresses funny, is passionate about the things he believes, and occasionally slips into political views that tend to be less accepted by "learned" people. So Dryden gets credited with "the best hockey book ever written," and Cherry gets scorn.

Coincidentally, Cherry has books of hockey stories too (as dictated to sportswriter Al Strachan); I just finished reading his second one. The styles, as you can imagine, are different. Dryden shrouds each locker room conversation in a cloud of profundity and mood; Cherry tells it in his straight-ahead style. But Cherry has something to say, and a point to make, and the stories are actually interesting. Dryden's got a couple of good ones -- like Steve Shutt urinating into a cup, adding Coke for color, and waiting for one of his teammates to come by and drink it -- but most of them are, well, not that engrossing. Like this one:

Amid the business of getting ready for practice, there is talk of beer.

"Calisse, you see the paper?" [Rejean] Houle moans. "Beer's goin' up sixty-five cents a case. Sixty-five cents!"

His words bring a grumble of memory.

"Shit, yeah," says a mocking voice, "the only thing should go up is what they pay fifteen-goal scorers, eh, Reggie?"

There is laughter this time. Across the room, Guy Lapointe stares at the ceiling, lost in thought. Suddenly he blurts, "That's it, that's it. No more drinkin'."

There is loud laughter.

"Hey, Pointu," Steve Shutt says, "ya just gotta learn to beat the system -- drink on the road."

That one, it seems safe to say, wouldn't have made it into Cherry's book. And, as far as actual hockey content goes, Cherry has Dryden beat by a mile. I opened Cherry's book to a random page, 74, where there's a thing about fighting. It's too long to quote entirely, but here's what Cherry tells me in five paragraphs:

-- There is no rush in the world like when you fight.

-- When players get older, they get a conscience and start to hesitate, and they have a tough time. That's why you see few older players fighting.

-- Fighters' hands suffer serious damage. "You wouldn't believe the hands on Joey Kocur ... it looks like he's had a ping-pong ball implanted under each knuckle."

-- The advent of helmets and visors have made hand injuries much more common, so fighters will remove their helmets as a show of respect. "I love it when I see one guy who's have trouble getting the strap loose on his helmet, and the other guy gives him time while he gets it off. ... That's honour! I love those guys."

No exaggeration: I learned more in that half-page than probably in 50 pages of Ken Dryden. Now, I'm not saying Don Cherry is always right, or even mostly right, or that you have to agree with him, or that he's particularly eloquent. But, he does try to tell you something about hockey. He may be wrong about some of the things he believes, like Joe Morgan or Harold Reynolds, or any other commentator in any other sport. But, geez, at least he says things about hockey!

As much as people love the Ken Drydens of the world, it's the Don Cherrys who actually can teach you things. I mean, they're not always right: for every non-Dryden who believes something correctly and passionately, there'll be many other non-Drydens who believe something different, incorrectly but just as passionately. So I'm not saying that when you find a Don Cherry, you should immediately become a mindless follower. I'm just saying that if you actually want to know how things are, you should start with the Cherrys and then go with your brain. Whatever the subject, hockey or otherwise, you'll never learn anything unless you stick with the people who are seriously trying to tell you something -- not just the people who sound the best.

------

No offense to Ken Dryden. I'm not saying that he doesn't know anything about hockey, just that he chose not to tell us too much about it. He actually wrote a pretty decent book, just one that's not as much to my taste as it could have been. And Dryden is under no obligation to write the kind of book that I like to read.

Still, "The Game" is nowhere near the best hockey book ever written. It's just the most poetic.

A "Grantland" article on Moneyball effects

2011-12-09T14:21:00.009-05:00

Here's a baseball salary article at Grantland, by economists Tyler Cowen and Kevin Grier. It’s a strange one ... the impression I get is that is that the authors are just going on the basics of the "Moneyball" story, but don’t really follow baseball discussions very much. And so some of their arguments are obviously behind the curve.

For instance, they talk about how closers used to be paid inefficiently, but aren't any more, except by free-spending teams like New York:

"This year, the Yankees' Mariano Rivera was ranked fifth in total saves with 44. At a salary of $14.9 million, that works out to be a hefty $338,600 per save. The four closers ranked ahead of him averaged 46.5 saves and a salary of $2.9 million, or $63,771 per save — quite the bargain."

The problem here is obvious to almost any serious baseball fan: closers aren’t normally evaluated by the number of saves, which is mostly a function of the opportunities the team provides. Rather, and like any other member of the roster, the closer is paid according to how many wins he can contribute to the team's record, as compared to a replacement player. For Rivera to be worth $15 million, he has to contribute about three extra wins (at a going rate of $4.5 million per win). Which means, basically, he has to blow three fewer saves, given his opportunities. Or, rather, he has to be *expected* to blow three fewer saves; there's still a lot of randomness there.

But Cowen and Grier don't mention randomness at all. And their only reference to blown saves is in one sentence that mentions the Twins' Joe Nathan and Matt Capps, who blew 12 saves out of 41 opportunities.

Another thing, too, is that the article doesn't mention one big difference between Rivera and the others: Rivera is a free agent, while young players like Neftali Feliz can be paid whatever the team wants. The Yankees might prefer Feliz to Rivera, but that’s not a choice they have open to them.

It's not a new "Moneyball" discovery that "slaves" make less money than established free-agent stars ... but the article seems to imply that teams don’t realize that the $400,000 stopper can be just as valuable, for the money, as the $15,000,000 stopper.

To me, it looks like the problem is that if you don’t know baseball that well, you tend to overrate the “Moneyball” possibilities, because that’s the story that you’ve heard the most.

-----

The authors then go on to say:

"The best-known Moneyball theory was that on-base percentage was an undervalued asset and sluggers were overvalued. At the time, protagonist Billy Beane was correct. Jahn Hakes and Skip Sauer showed this in a very good economics paper. From 1999 to 2003, on-base percentage was a significant predictor of wins, but not a very significant predictor of individual player salaries. That means players who draw a lot of walks were really cheap on the market, just as the movie narrates."

The authors imply that “walks were really cheap on the market,” means that the A’s had a huge hole to exploit.

But ... even if walks were indeed “really cheap,” it would still be a small hole. Walks are a significant part of a player’s value, but still in the sense of a small edge, not a huge one. Suppose teams valued walks at only half their actual value. If you can pick up a player with 60 walks, for the price of 30, you’ll gain about 10 runs, or one win. Not a big deal.

Of course, if you can do that nine times, that’s nine free wins. But the A’s didn’t. In 2002, they walked 609 times, third in the league. But that was only 157 more walks than Baltimore, second-worst in the league. If 157 was the number of walks they got at half-price, that’s still only two or three wins.

You could choose, instead, to compare the A’s to the 2002 Tigers, who walked only 363 times. That would be completely unrealistic, in my view, to assume the A’s would have been as bad as one of the worst recent teams ever. But if you do, you *still* only gain four wins.

----

The authors also put too much faith in the Hakes/Sauer paper. As I wrote a few years ago, it seems to me that the paper has a few problems, and I don’t think it shows what it purports to show.

The study found a huge increase in the correlation between salary and OBP between 2003 (when the "Moneyball" book was released) and 2004. The numbers for 2004 almost exactly matched the actual value of a walk, so the authors concluded that the market became efficient in the off-season, and teams wised up after reading the book..

But that conclusion doesn’t make sense. Since only a small percentage of players got new contracts between 2003 and 2004, for the overall average to move so much, the market would have had to overcompensate for walks by double, or triple their real value! That doesn’t sound like a reasonable possibility, and it’s certainly not consistent with GMs now learning to be efficient.

-----

Finally, on the subject of correlation:

"Here's something funny about the Moneyball strategy: It is bringing us a world where payroll matters more and more. Spotting undervalued players boosts their salaries and makes money more important for the general manager; little did Billy Beane know that in the long run he would be strengthening the hand of the large home-market teams, such as the Yankees. From 1986 to 1993, payroll explained 2.2 percent of the variation in team winning percentage, and that meant spending more money yielded little return in terms of quality on the field. In the 2004 to 2006 seasons, after the Moneyball revolution was under way, payroll explained 27.1 percent of the variation in team winning percentage, which means a stronger reason to spend more."

I've written about this before, and Tango’s written about it several times: a higher r-squared does NOT necessarily mean money is more important in buying wins. Rather, the r-squared is a combination of:

1. the extent to which money can actually buy wins;
2. the extent to which teams differ in spending, in real-life.

When the authors say, "spending more money yielded little return," they seem to be assuming it’s all the first thing, when it might be all the second thing.

As an example, take dueling, where two people go out at dawn, draw weapons, and one of them kills the other. Back when it was legal, dueling would explain a lot of the variation in death rates of people who didn’t like each other. Now that it’s illegal, it explains zero.

However, the fact that the r-squared dropped doesn’t mean that dueling is any less dangerous than it used to be (point 1) -- it just means that people no longer vary in how often they get killed in duels (point 2).

The same thing could be happening here. I did a Google search and found an article (.pdf) that gives some team payroll data for the period the article covers. From Table 1, the article shows that from 1985 to 1990, fourth quartile teams (the 25% of teams with the highest payrolls) outspend the first quartile teams by only about 2 to 1. From 1998 to 2002, the ratio jumped to 3 to 1. The paper only covers to 2002, but a glance at later numbers seems to show around 2.5 to 1 (but up to 3.1 to 1 for the 2011 season).

This is evidence that at least *some* of the difference is probably caused by teams being willing to spend more.

I may be unfair to the authors here ... that might be partly what they’re saying. If I read them right, they’re saying that, armed with "Moneyball" concepts, teams are realizing they can buy wins cheaper by evaluating players more accurately (1) -- and, that teams are therefore more likely to vary in how much they pay when they know it’s money well spent (2).

But ... well, I think these effects are pretty small. As I argued, walks are a small part of the overall equation, even if they were undervalued by half (which itself is probably an overestimate). It’s not like, in 1990, teams were paying Jose Oquendo as much as Wade Boggs. To be sure, teams weren’t perfect in evaluating players -- but they were still reasonably good. Any improvement since then has to be relatively small, at the margins.

So, the idea that teams would say, "hey, we can now evaluate players slightly more accurately, so let’s go on a spending spree" doesn’t seem all that plausible.

------

What actually *did* happen to tighten the relationship between payroll and wins? As usual, you guys probably know better than I do. I’ll give you my guess anyway, which is that it’s a combination of a bunch of things:

1. It became more "socially acceptable" for teams to pay big money to free agents. Remember, 1985 to 1990 includes the collusion year, and there was probably a significant amount of pressure to keep spending down. That pressure was probably more significant in discouraging headline-grabbing salaries, rather than routine signings, so maybe a player who was twice as valuable wouldn’t be able to sign for twice as much. That would help keep the correlation between salary and success low.

2. When baseball revenues exploded, they grew more in some cities than others. That meant that marginal wins would be extremely valuable to the Yankees, but not so much to the Pirates. That increased the variation in team spending, which pushed up the r-squared.

3. Teams got smarter, in line with Cowen and Grier’s theory. But I think that was a small part of what happened. Also, I’d guess that a lot of improvement in that regard would have happened well before Moneyball, as Bill James’ discoveries got around a bit. Conventional wisdom denies that baseball executives put any faith in what Bill James had to say, but ... I dunno, good ideas tend to get noticed, even if people say they don’t believe in them. Also, Bill James’ ideas showed up early in arbitration hearings, which affected the teams’ bottom lines pretty much immediately.

4. Randomness. In a team payroll to wins regression, Cowen and Grier give an r-squared of .022 for 1986 to 1993.

(By the way, I assume Cowen and Grier's regression adjusted for payroll inflation ... salaries more than doubled between 1986 and 1993. If they didn't adjust, that might explain the low correlation.)

I wonder if that .022 might just be an outlier. Here are equivalent numbers from Berri/Schmidt/Brook in "The Wages of Wins," page 40:

Wages of Wins:

1988 to 1994: r-squared = .062, r = .25
1995 to 1999: r-squared = .325, r = .57
2000 to 2005: r-squared = .176, r = .42

Cowen/Grier:

1986 to 1993: r-squared = .022, r = .15

The numbers sure do move around a lot! It probably doesn’t take much to knock the correlation down: you need a few teams to get lucky in exceeding their talent, and a few teams to get lucky and get some good slaves and arbs. Maybe I’ll try a simulation and see how common a .022 might actually be.

Transparent studies are better, even if they're less rigorous

2011-12-06T14:29:00.003-05:00

In 1985, Orioles manager Joe Altobelli claimed that power pitchers do better than finesse pitchers in cold weather. In the 1986 Baseball Abstract, Bill James wanted to check whether that was true.

So, he went through baseball history, and found sets of pitchers with exactly the same season W-L record, but where one was a power pitcher and one was a finesse pitcher. For example, Tom Seaver, a power pitcher, went 22-9 in 1975. He got paired with Lary Caldwell, a finesse pitcher, who went 22-9 in 1978.

Bill found 30 such pairs of pitchers. So, he had two groups of 30 pitchers, each with identical 539-345 (.610) records overall.

He compared the two groups in April, the cold-weather month. As he put it, "Altobelli was dead wrong. He couldn't have been more wrong." It turned out that the power pitchers were only 49-51 (.490) in April, while the finesse pitchers were 63-37 (.630). That's exactly opposite to what Altobelli had thought.

------

Nice study, right? I love it ... it's one of my favorites. But it wasn't very "sophisticated" in terms of methodology. For instance, It didn't use regression.

Should it? Well, I suppose you could make that argument. With regression, the study can be more precise. Bill James used only season W-L record in his dataset, but in your regression, you could add a lot more things: ERA, strikeouts, walks. You could include dummy variables for season, park, handedness, and lots of other things. And, of course, you wouldn't be limited to only 30 pairs of pitchers.

And you'd get a precise estimate for the effect, with confidence intervals and p-values.

But ... in my opinion, that would make it a WORSE study, not better. Why? One big reason: Bill's study is completely transparent, and understandable.

Take the four paragraphs above, and show them to a friend who doesn't know much about statistics. That person, if he's a baseball fan, will immediately understand how the study worked, what it showed, and what it means.

On the other hand, if you try to explain the results of a regression, he won't get it. Sure, you could explain the conclusions, and what the coefficient of walks and strikeouts mean, and so on. And he might believe you. But he won't really get it.

With the easy method, anyone can understand what the evidence means. With regression, they have to settle for understanding someone else's explanation of what the evidence means.

Reading Bill's study is like being an eyewitness to a crime. Reading the results of the regression is like hearing an expert witness testify what happened.

------

Now, you may object: why is that so important? After all, if it takes a sophisticated method to uncover the truth, well, that's what we have to do. Sure, it's nice if the guy on the street can be an eyewitness and understand the evidence, but that's not always possible. If we limited our research studies to methods that were intuitive to the layman, we'd never learn anything! Physics is difficult, but gives us cars and airplanes and electronics and nuclear energy. If it takes a little bit of effort and education to be able to do it, then that's the price we pay!

To which I have two responses. The first one is, that, actually, I agree. I'm not saying that we should limit our research to *only* studies that laymen understand better. I'm just saying that it's preferable *when it's possible*.

The second response, though, is: it's not just laymen I'm talking about. It's also sophisticated statisticians and sabermetricians. You, and me, and Tango, and Bill James, and JC Bradbury, and David Berri, and Brian Burke, and all those guys.

Because, the truth is, a regression study is not transparent to ANYBODY. I mean, I've read a lot of regressions in the last few years, and I can tell you, it takes a *lot* of work to figure out what's going on. There are a lot of details, and, even when the regression is simple, there's a lot of work do to in the interpretation.

For instance, a few paragraphs ago, I gave a little explanation of how a regression might work for Bill's study. And, suppose, making some numbers up, I get a coefficent of -.0007 per strikeout, and -.0003 per walk. What does that mean?

Well, you have to think about it. It means, for instance, that if Tom Seaver had 50 extra strikeouts and 20 extra walks than pitcher X, his April winning percentage will be .041 worse, all things being equal, than X. But ... it takes a while, and you have to do it in your head.

But wait! Not every pitcher in the study had the same number of innings pitched. So Tom Seaver's 50 extra strikeouts, do I have to convert that to a fixed number of innings? What did the study use? Now I have to go back and check.

And how do I interpret the results? Am I sure they really apply to all pitchers? I mean, suppose there's a pitcher like Seaver, who strikes out a lot of guys, but has better control, so he walks 30 fewer guys. Should I really assume that he'll do better in April than Seaver, since he's "less" of a power pitcher in that dimension?

Also, wait a sec. The regression included season W-L record, but that *includes* the April W-L record that it was trying to predict. That will throw off the results, won't it? Maybe the regression should have used only May to October. Or maybe it did? Now I have to go check that.

And what if it didn't, and there were a bunch of pitchers that pitched only one game after April? Will that throw off the results? If the confidence intervals suspect, are the coefficients suspect too?

I could go on ... there are a thousand issues that affect the interpretation of the regression's results. And it's impossible for any one person, even the best statistician in the world, to hold all thousand in his head at once.

------

You could now come back with another argument: "OK, the regression is harder to interpret, but we can just do that interpretation. Indeed, that's the duty of the researcher. When you write up a study, your duty isn't just to do a regression and report the results. It's to figure out what the results mean, and check that everything's been done right. Also, that's what the peer reviewers are for, and the journal editors."

To which the most obvious reply is: it's simply not true that peer reviewing makes sure everything is correct. There are loads and loads of problems with actual interpretation of regressions, some of which I've written about here. There was the paper that forgot to hold everything constant when interpreting a coefficient. There was the paper that decided that X picks ahead was worth zero, but drafting 2X picks ahead was significant. There was the paper that decided that a low "coefficient of determination" meant that the effect was insignificant, regardless of what the coefficient showed. And so on.

And those are the easy ones. I mean, sure, I'm no sophisticated peer reviewer with a Ph.D. who looks through a hundred of these a month, but I do have a decent basic idea of how regressions work and how baseball works. But, for some of these regressions, it took me a long time, measured in hours, to figure out what they actually were doing and (in many cases) why the results didn't really mean what the author thought they meant. It's not that you need the right kind of expertise ... it's that every case is different, and there's no formula. To figure out what a result actually means, you have to look at everything: where the data comes from, how it interacts, what is really being measured, what the coefficients mean, and, especially, if the model is realistic and if other models give different results.

As I've said before, regression is easy. Interpreting the regression is hard -- legitimately hard. And, unlike other hard problems, you don't know when or whether you've found the right answer. You can spend days looking at it, and you might still be missing something.

------

Which, of course, means that the simple method is more likely to be correct than the complicated method: if we understand a study, we're much more likely to spot its flaws. If Bill James did something dumb in how he compiled his data, most of us will catch it. If Joe Regressor does something wrong in his complicated regression, it's likely that nobody will see it (unless it's in a hard science, in which case a plane will crash or a patient will die or something).

And, of course, there's still the advantage that the simple study is easier to understand. That's an advantage even if the regression study is absolutely 100% correct. If you can see the answer in four paragraphs of arithmetic, that's better than if it takes ten pages of regression notation.

Which advantage is more important? Actually, it looks like the first one is more important ... but, you know, the second one can make a strong case.

That Bill James study that I mentioned earlier ... if you have a copy of the 1986 Abstract handy, you should go read it. It's on page 134. It's only two pages long, and easy reading, like most of Bill's prose.

If you've read it, I'd say that, right now, you KNOW the answer to Bill's question. I don't mean you know 100% that he's right, and what the answer is ... I'm saying that you know, almost 100%, the evidence and the logic. You may or may not think the results are conclusive -- for instance, I'd like to see a larger sample size -- but, either way, it's your own decision, not Bill's.

That is: even though Bill did the study, you instantly absorb the answer. You don't have to trust Bill about it. Well, you have to trust that he aggregated the data properly, and didn't cheat in what pitchers he chose. But you don't have to trust Bill's judgment, or Bill's knowledge of baseball, or Bill's interpretation. His interpretation will become yours: you'll see what he did, and understand it well enough that if someone challenges it, you can defend it. Bill's study is so transparent, that, after you read it, you understand the answer as well as Bill did, probably about as well as anyone can.

That's not true for a regression study. Often, for many readers (myself included), the explanation is impenetrable. The references for the methods refer you to textbooks that are hard to find and technical, and, usually, there's no discussion of what the results mean, other than just a surface reading of the coefficients. If you want to truly understand what's going on, you have to read the study, and read critically. Then you have to read it again, filling in the missing pieces. Then you have to look at the tables, and back to the text, then to the model. And then you still probably have to read it again.

And all this is assuming you already know something about how regression works. If you don't, you'll just have no idea.

So the difference is: if the study is simple, you know what it means. If the study is complicated, you don't know anything. You have to trust the person that wrote the study.

It's night and day: knowing versus not knowing. With the Bill James study, you know it's true. With a regression study, you just believe it's true.

------

To go a bit off topic ...

A lot of the factual things we think we know, and will passionately defend, we don't really know, except from what other people tell us. I bet everyone reading this has a strong opinion on creation vs. evolution, or whether global warming is real or a hoax, or whether 9/11 was or was not partly an inside job by the US government.

Take evolution. Let's suppose you believe in evolution, and you think creationism is not true, just wishful thinking by creationists. I think most of my friends fall into this category, and I've read surveys that say most Americans generally believe this.

But if that's you, can you really say that you KNOW it? You don't, probably. I bet you that for most of you (myself included), if someone asked you for examples of actual evidence for evolution, we'd have nothing. Seriously, zero. I couldn't even give a half-hearted attempt at a single sentence of why and how we know that evolution actually happened.

Sure, I believe evolution happened, but not because of any actual evidence. I believe it for secondhand reasons. I believe it happened because I know that scientists, serious researchers, have looked at the evidence, and that it's strong evidence. What I *do* think I know, from dealing (at arm's length) with teachers and authors and scientists and journalists, is that it's very, very unlikely that the worldwide supply of scientists, working independently and jockeying for discoveries and status and publications, and being so steeped in scientific method, and competing in an open marketplace of ideas, could have deluded themselves into misinterpreting so much evidence over so many decades.

So, you know, I can't say I know evolution is true. But now I DO know there is strong evidence that power pitchers do not outperform in April.

That's the beauty of the Bill James-type study, the one that lays it all out for you without using fancy techniques. It gives you a completely different feeling, the feeling that you actually know something, instead of just believing it from hearsay.

And that, to me, is an important part of what science is all about.

Why it's hard to estimate small effects

2011-11-30T10:46:00.011-05:00

Here's a great 2009 paper (.pdf) by Andrew Gelman and David Weakliem (whom I'll call "G/W"), on the difficulty of finding small effects in a research study.

I'm translating to baseball to start.

-----

Let's suppose you have someone who claims to be a clutch hitter. He's a .300 hitter, but, with runners on base, he claims to be a bit better.

So, you say, show us! You watch his 2012 season, and see how well he hits in the clutch. You decide in advance that if it's statistically significantly different from .300, that will be evidence he's a clutch hitter.

Will that work? No, it won't.

Over 100 AB, the standard deviation of batting average is about 46 points. To find statistical significance, you want 2 SD. That means to convince you, the player would have to hit .392 in the clutch.

The problem is, he's not a .392 hitter! He, himself, is only claiming to be a little bit better than .300. So, in your study, the only evidence you're willing to allow, is evidence that you *know* can't be taken at face value.

Let's say the batter actually does meet your requirement. In fact, let's suppose he exceeds it, and hits .420. What can you conclude?

Well, suppose you didn't know in advance that you were looking for small effect. Suppose you were just doing a "normal" paper. You'd say, "look, he beat his expectation by 2.6 SD, which is statistically significant. Therefore, we conclude he's a clutch hitter." And then you write a "conclusions" section with all the implications of having a .420 clutch hitter in your lineup.

But, in this case, that would be wrong, because you KNOW he's not a .420 clutch hitter, even though that's what he hit and you found statistical significance. He's .310 at best, maybe .320, if you stretch it. You KNOW that the .420 was mostly due to luck.

Still ... even if you can't conclude that the guy is truly a .420 clutch hitter, you SHOULD be able to at least conclude that he's better than .300 right? Because you did get that statistical significance.

Well ... not really, I don't think. Because, the same evidence that purports to show he's not a .300 hitter ALSO shows he's not a .320 hitter! That is, .420 is also more than 2 standard deviations from .320, which is the best he possibly could be.

What you CAN do, perhaps, is compare the two discrepancies. .420 is 2.6 SDs from .300, but only 2.2 SDs from .320. That does appear to make .320 more likely than .300. In fact, the probability of a .320 hitter going 42-for-100 is almost five times as high as the probability of the .300 hitter going 42-for-100.

But, first, that's only 5 in 6. Second, that ignores the fact that there are a lot more .300 hitters than .320 hitters, which you have to take into account.

So, all things considered, you should know in advance that you won't be able to conclude much from this study. The sample size is too small.

-------

That's Gelman and Weakliem's point: if you're looking for a very small effect, and you don't have much data, you're ALWAYS going to have this problem. If you're looking for the difference between .300 and .320, that's a difference of 20 points. If the standard error of your experiment is a lot more than 20 points ... how are you ever going to prove anything? Your instrument is just too blunt.

In our example, the standard error is 46 points. To find statistical significance, you'd have to observe an effect of at least 92 points! And so, if you're pretty sure clutch hitting talent is less than 92 points, why do the experiment at all?

But what if you don't know if clutch hitting talent is less than 92 points? Well, fine. But you're still never going to find an effect less than 92 points. And so, your experiment is biased, in a way: it's set up to only find effects of 92 points or more.

That means that if the effect is small, no matter how many scientists you have independently searching for it, they'll never find it. Moreover, they will frequently find a LARGE effect.

No matter what happens, the experiment will either be wrong too high, or wrong too low. It is impossible for it to be accurate for a small effect. The only way to find a small effect is to increase the sample size. But even then, that doesn't eliminate the problem: it just reduces it. No matter what your experiment, and how big your sample size, if the effect your looking for is smaller than 2 SDs, you'll never find it.

That's G/W's criticism. It's a good one.

-------

G/W's example, of course, is not about clutch hitting. It's about a previously-published paper, which found that good-looking people are more likely to produce female offspring than male offspring. That study found an 8 percentage point difference between the nicest-looking parents and the worst-looking parents -- 52 percent girls vs. 44 percent girls.

And what G/W are saying is, that 8 point difference is HUGE. How do they know? Well, it's huge as compared to a wide range of other results in the field. Based on the history of studies on birth sex bias, two or three points is about the limit. Eight points, on the other hand, is unheard of.

Therefore, they argue, this study suffers from the "can't find the real effect" problem. The standard error of the study was over 4 points. How can you find an effect of less than 3 points, if your standard error is 4 points? Any reasonable confidence interval will cover so much of the plausible territory, that you can't really conclude anything at all.

Gelman and Weakliem don't say so explicitly, but this is a Bayesian argument. In order to make it, you have to argue that the plausible effect is small, compared to the standard error. How do you know the plausible effect is small? Because of your subject matter expertise. In Bayesian terms, you know, from your prior, that the effect is most likely in the 0-3 range, so any study that can only find an 8-point difference must be biased.

Every study has its own limits of how the standard error compares to the expected "small" effect. You need to know what "small" is. If a clutch hitting study was only accurate to within .0000001 points of batting average ... well, that would be just fine, because we know, from prior experience, that a clutch effect of .0000002 is relatively plausible. On the other hand, if it's only accurate to within .046, that's too big -- because a clutch effect of .092 is much too large to be plausible.

It's our prior that tells us that. As I've argued, interpreting the conclusions of your study is an informal Bayesian process. G/W's paper is one example of how that kind of argument works.

--------

Hat tip: Alex Tabarrok at Marginal Revolution

Why p-value isn't enough, reiterated

2011-11-28T18:48:00.013-05:00

Question 1:

People are routinely tested for disease X, which 1 in 1000 people have overall. It is known that if the person has the disease, the test is correct 99% of the time. If the person does not have the disease, the test is also correct 99% of the time.

A patient goes to his doctor for the test. It comes out positive.

What is the probability that the patient has the disease?

Question 2:

Researchers routinely run studies to test unexpected hypotheses (such as: can outside prayer help cure disease?), of which 1 in 1000 tend to be true overall. It is known that if a hypothesis is true, a study correctly finds statistical significance 99% of the time. If the hypothesis is false, the study correctly finds NO statistical significance 99% of the time.

A researcher tests one such unexpected hypothesis. He finds statistical significance.

What is the probability that the hypothesis is true?

--------

Hat Tip: Inspired by Jeremy's last paragraph of comment #25, here.

--------

P.S. Answer to question 1 (very slightly modified question, but the same answer) at my previous post, here.

Research conclusions have to be bayesian

2011-11-23T10:10:00.009-05:00

The last couple of posts here have been about interpreting the results of statistical studies. I argued that the statistical method itself might be just fine, but the *interpretation* of what it means, the conclusions you draw about real life, require an argument. That is, you can get the regression right, but the conclusions wrong, because the conclusions call for argument and judgment.

Or, as some commenters have substituted, "intuition" and "subjectivity". Those are negative things, in academic circles. Objectivity is the ideal, and the idea that the reliability of a work of scholarship depends on a subjective evaluation of the author's judgment doesn't seem to be something that people like.

But, I think it absolutely has to follow. If you find a connection between A and B, how do you know if it's A that causes B, or B that causes A, or if it's all just random? That's something no statistical analysis can tell you. By definition, it calls for judgment, doesn't it? At least a little bit. Recall the recent (contrived) study that showed that listening to kids' music is linked to being physically older. Nobody would conclude that the music MAKES you older, right? But that's not a result of the statistical analysis -- it's a judgment based on outside knowledge. An easy, obvious judgment, but a judgment nonetheless.

It occurred to me that this judgment, that takes you from regression results to conclusions, is really an informal Bayesian inference. I don't think this is a particularly novel insight, but it helps to make the issue clearer. My argument is this: first, even if you do a completely normal, ("frequentist") experiment, the step from the results to the conclusions HAS to be Bayesian. And, more importantly, because Bayesian techniques sometimes require judgment, and are therefore not completely objective, the convention has been to avoid such judgment in academic papers. Therefore, these studies have locked themselves in to a situation in which they have to suspend judgment, and use strict rules, which sometimes lead to wrong -- or seemingly absurd -- answers.

OK, let me start by explaining Bayesianism, as I understand it, first intuitively, then in a baseball context. As always, real statisticians should correct me where I got it wrong.

----------

Generally, Bayesian is a process by which you refine your probability estimate. You start out with whatever evidence you have which leads you to a "prior" estimate for how things are. Then, you get more evidence. You add that to the pile, and refine your estimate by combining the evidence. That gives you a new, "posterior" estimate for how things are.

You're a juror at a trial. At the beginning of the trial, you have no idea whether the guy is guilty or not. You might think it's 50/50 -- not necessarily explicitly, but just intuitively. Then, a witness comes up that says he saw the crime happen, and he's "pretty sure" this is the guy. Combining that with the 50/50, you might now think it's 80/20.

Then, the defense calls the guy's boss, who said he was at work when the crime happened. Hmmm, you say, that sounds like he couldn't have done it. But there's still the eyewitness. Maybe, then, it's now 40/60.

And so on, as the other evidence unfolds.

That's how Bayesian works. You start out with your "prior" estimate, based on all the evidence to date: 50/50. Then, you see some new evidence: there's an eyewitness, but the boss provides an alibi. You combine that new evidence with the prior, and you adjust your estimate accordingly. So your new best estimate, your "posterior," is now 40/60.

---------

That's an intuitive example, but there is a formal mathematical way this works. There's one famous example, which goes like this:

People are routinely tested for disease X, which 1 in 1000 people have overall. It is known that if the person has the disease, the test is correct 100% of the time. If the person does not have the disease, the test is correct 99% of the time.

A patient goes to his doctor for the test. It comes out positive. What is the probability that the patient has the disease?

If you've never seen this problem before, you might think the chance is pretty high. After all, the test is correct at least 99% of the time! But that's not right, because you're ignoring all the "prior" evidence, which is that only 1 in 1000 people have the disease to begin with. Therefore, there's still a strong chance that the test is a false positive, despite the 99 percent accuracy.

The answer turns out to be about 1 in 11. The (non-rigorous) explanation goes like this: 1000 people see the doctor. One has the disease and tests positive. Of the other 999 who don't have the disease, about 10 test positive. So the positive tests comprise 10 people who don't have the disease, and 1 person who does. So the chance of having the disease is 1 in 11.

Phrasing the answer in terms of Bayesian analysis: The "prior" estimate, before the evidence of the test, is 0.001 (1 in 1000). The new evidence, though, is very significant, which means it changes things a fair bit. So, when we combine the new evidence with the prior, we get a "posterior" of 0.091 (1 in 11).

If that still seems counterintuitive to you, think of it this way: if the test is 99% positive, that's 1 in 100 that it's wrong. That's low odds, which makes you think the test is probably right! But ... the original chance of having the disease is only 1 in 1000. Those are even worse odds. The prior of 1/1000 competes with the new evidence of 1/100. Because the new number (test being wrong) is more likely than the old number (no disease), the odds are skewed to the test being wrong: odds of 10:1 that the test is wrong, compared to the patient having the disease.

Another way to put it: the less likely the disease was to start with, the more evidence you need to overcome those low odds. 1/100 isn't enough to completely overcome 1/1000.

(Perhaps you can see where this will be going, which is: if a research study's hypothesis is extremely unlikely in the first place, even a .01 significance level shouldn't be enough to overcome your skepticism. But I'm getting ahead of myself here.)

---------

Let's do an oversimplified baseball example. At the beginning of the 2011 baseball season, you (unrealistically) know there's a 50% chance that Albert Pujols' batting average talent will be .300 for the season, and a 50% chance that his batting average talent will be .330. Then, in April, he goes 26 for 106 (.245). What is your revised estimate of the chance that he's actually a .300 hitter?

You start with your "prior" -- a 50% chance he's a .300 hitter. Then, you add the new evidence: 26 for 106. Doing some calculation, you get your "posterior." I won't do the math here, but if I've got it right, the answer is that now the chance is 80% that Pujols is actually a .300 hitter and not a .330 hitter.

That should be in line with your intuition. Before, you thought there was a good chance he was a .330 hitter. After, you think there's still a chance, but less of a chance.

We started thinking Pujols was still awesome. Then he hit .245 in April. We thought, "Geez, he probably isn't really a .245 hitter, because we have many years of prior evidence that he's great! But, on the other hand, maybe there's something wrong, because he just hit .245. Or maybe it's just luck, but still ... he's probably not as good as I thought."

That's how Bayesian thinking works. We start with an estimate based on previous evidence, and we update that estimate based on the new evidence we add to the pile.

--------

Now, for the good part, where we talk about academic regression studies.

You want to figure out whether using product X causes cancer. You do a study, and you find statistical significance at p=0.02, and the coefficient says that using product X is linked with a 1% increased chance of cancer. You are excited about your new discovery. What do you put in the "conclusions" section of your paper?

Well, maybe you say "this study has found evidence consistent with X causing cancer." But that isn't helpful, is it? I mean, you also found evidence that's consistent with X *not* causing cancer -- because, after all, it could have just been random luck. (A significance level of .02 would happen by chance 1 out of 50 times.)

Can you say, "this is strong evidence that X causes cancer?" Well, if you do, it's subjective. "Strong" is an imprecise, subjective word. And what makes the evidence "strong"? You better have a good argument about why it's strong and not weak, or moderate. The .02 isn't enough. As we saw in the disease example, a positive test -- which is equivalent to a significance level of .01, since a positive test happens only 1 in 100 times -- was absolutely NOT strong evidence of having the disease. (It meant only a 1 in 11 chance.)

Similarly, you can't say "weak" evidence, because how do you know? You can't say anything, can you?

It turns out that ANY conclusion about what this study means in real life has to be Bayesian, based not just on the result, but on your PRIOR information about the link between cancer and X. There is no conclusion you can draw otherwise.

Why? Well, it's because the study has it backwards.

What we want to know is, "assuming the data came up the way it did, what is the chance that X causes cancer?"

But the study only tells us the converse: "assuming X does not cause cancer, what is the chance that the data would come up the way it did?"

The p=0.02 is the answer to the second question only. It is NOT the answer to the first question, which is what we really want to know. There is a step of logic required to go from the second question to the first question. In fact, Bayes' Theorem gives us the equation for finding the answer to the first question given the second. That equation requires us to know the prior.

What the study is asking is, "given that we got p=0.02 in this experiment, what's the chance that X causes cancer?" Bayes' Theorem tells us the question is unanswerable. All we can answer is, "given that we got p=0.02 in this experiment, what is the chance that X causes cancer, given our prior estimate before this experiment?"

That is: you CANNOT make a conclusion about the likelihood of "X causes cancer" after the experiment, unless you had a reliable estimate of the likelihood of "X causes cancer" BEFORE the experiment. (In mathematical terms, to calculate P(A|B) from P(B|A), you need to know p(B) and p(A).)

Does this sound wrong? Do you think you can get a good intuitive estimate just from this experiment alone? Do you feel like the .02 we got is enough to be convincing?

Well, then, let me ask you this: what's your answer? What do you think the chance is that X causes cancer?

If you don't agree with me that there's no answer, then figure out what you think the answer is. You may assume the experiment is perfectly designed, the sample size is adequate, and so on.

If you don't have a number -- you probably don't -- think of a description, at least. Like, "X probably causes cancer." Or, "I doubt that X causes cancer." Or, "by the precautionary principle, I think everyone should avoid X." Or, "I don't know, but I'd sure keep my kids away from X until there's evidence that it's safe!"

Go ahead. I'll leave some white space for you. Get a good intuitive idea of what your answer is.

(Link to Jeopardy music while you think (optional))

OK. Now, I'm going to tell you: product X is a bible.

Does that change your mind?

It should. Your conclusion about the dangers of X should absolutely depend on what X is -- more specifically, what you knew about X before. That is, your PRIOR. Your prior, I hope, had a probability of close to 0% that a Bible can cause cancer. That's not just a wild-ass intuition. There are very good, rational, objective reasons to believe it. Indeed, there is no evidence that the information content of a book can cause cancer, and there is no evidence or logic that would lead you to believe that bibles are more carcinogenic than, say, copies of the 1983 Bill James Baseball Abstract.

Call this "intuition" or "subjectivity" if you want. But if you decide not to use your own subjective judgment, what are you going to do? Are you going to argue that bibles cause cancer just to avoid having to take a stand?

I suppose you can stop at saying, "this study shows a statistically significant relationship between bible use and cancer." That's objectively true, but not very useful. Because the whole point of the study is: do bibles cause cancer? What good is the study if you can't apply the evidence to the question?

--------

You could do the Bayesian approach thing more formally. That's what researchers usually mean when they talk about "Bayesian methods" -- they mean formal statistical algorithms.

To do a Bayesian analysis, you need a prior. You could just arbitrarily take something you think is reasonable. "Well, we don't believe there's much of a chance bibles cause cancer, so we're going to assume a prior 99.9999% probability that there's no effect, and we'll split up the last remaining .0001 in a range between -2% and +2%." Now, you do the study, and recalculate your posterior distribution, to see if you now have enough evidence to conclude there's a danger.

If you did that, you'd find that your posterior distribution -- your conclusion -- was that the probability of no effect went down, but only from 99.9999% to 99.995%, or something. That would make your conclusion easy: "the evidence should increase our worry that bibles cause cancer, but only from 1 in a million to 1 in 20,000."

But, that Bayesian technique is not really welcome in academic studies. Why? Because that prior distribution is SUBJECTIVE. The author can choose any distribution he wants, really. I chose 99.9999%, but why not 99.99999% (which is probably more realistic)? The rule is that academic papers are required to be objective. If you allow the author to choose any prior he wants, based on his own intuition or judgment, then, first, the paper is no longer objective, and second, there is the fear that the author could get any conclusion he wanted just by choosing the appropriate prior.

So papers don't want to assume a prior. So instead of arguing about the chance the effect is real, the paper just assumes it's real, and takes it at face value. If X appears to increase cancer by 1%, and it's statistically significant, then the conclusion will assume that X actually *does* increase cancer by 1%.

That sounds like it's not Bayesian. But, in a sense, it is. It's exactly the result you'd get from a Bayesian analysis with a prior that assumes every result is equally likely. Yes, it's objective, because you're always using the same prior. But it's the *wrong* prior. You're using a fixed assumption, instead of the best assumption you can, just because the best assumption is a matter of discretion. You're saying, "Look, I don't want to make any subjective assumptions, because then I'm not an objective scientist. So I'm going to assume that bibles are just as likely to cause 1% more cancers as they are to cause 0% more cancers."

That's obviously silly in the bible case, and, when it's that obvious, it looks "objective" enough that the study can acknowledge it. But most of the time, it's not obvious. In those cases, the studies will just take their results at face value, *as if theirs is the only evidence*. That way, they don't have to decide if their result is plausible or not, in terms of real-life considerations.

Suppose you have two baseball studies. One says that certain batters can hit .375 when the pitcher throws lots of curve balls. Another says that batters gain 100 points on their batting average after the manager yells at them in the dugout. Both studies find exactly the same size effect, with exactly the same significance level of, say, .04.

Of the two conclusions, which one is more likely to be true? The curve ball study, of course. We know that some batters hit curve balls better than others, and we know some batters hit well over .300 in talent. It's fairly plausible that someone might actually have .375 talent against curve balls.

But the "manager yells at them" study? No way. We have a strong reason to believe it's very, very unlikely that batters would improve by that much just because they were yelled at. We have clutch hitting studies, that barely find an effect even when the game is on the line. We have lots of other studies that, even when they do find an effect, like platooning, find it to be much, much less than 100 points. Our prior for the "manager yelling is worth 100 points" hypothesis is so low that a .04 will barely move it.

Still ... I guarantee you that if these two studies were published, the two "conclusions" sections would not give the reader any indication of the relative real-life likelihood of the conclusions being correct, except by reference to the .04. In their desire to be objective, the two studies would not only fail to give shadings of their hypotheses' overall plausibility, but they'd probably just treat both conclusions as if they were almost certainly true. That's the general academic standard: if you have statistical significance, you're entitled to just go ahead and assume the null hypothesis is false. To do anything else would be "subjective."

But while that eliminates subjectivity, it also eliminates truth, doesn't it? What you're doing, when you use a significance level instead of an argument, is that you're choosing what's most objective, instead of what's most likely to be right. You're saying, "I refuse to make a judgment, and so I'm going to go by rote and not consider that I might be wrong." That's something that sounds silly in all other aspects of life. Doesn't it also sound silly here?

--------

So, am I arguing that academics need to start doing explicit Bayesian analysis, with formal mathematical priors? No, absolutely not. I disagree with that approach for the same reasons other critics do: it's too subjective, and too subject to manipulation. As opponents argue, how do you know you have the right prior? And how can you trust the conclusions if you don't?

So, that's why I actually prefer the informal, "casual Bayesian" approach, where you use common sense and make an informal argument. You take everything you already know about the subject -- which is your prior -- and discuss it informally. Then, you add the new evidence from your study. Then, finally, you conclude about your evaluation of the real-life implications of what you found.

You say, "Well, the study found that reading the bible is associated with a 1% increase in cancer. But, that just sounds so implausible, based on our existing [prior] knowledge of how cancer works, that it would be silly to believe it."

Or, you say, "Yes, the study found that batters hit 100 points better after being yelled at by their manager. But, if that were true, it would be very surprising, given the hundreds of other [prior] studies that never found any kind of psychological effect even 1/20 that big. So, take it with a grain of salt, and wait for more studies."

Or, you say, "We found that using this new artificial sweetener is linked to one extra case of cancer per 1,000,000 users. That's not much different from what was found in [prior] studies with chemicals in the same family. So, we think there's a good chance the effect is real, and advise caution until other evidence makes the answer clearer."

That's what I meant, two posts ago, where I said "you have to make an argument." If you want to go from "I found a statistically significant 4% connection between cancer and X," to "There is a good chance X causes cancer," you can't do that, logically or mathematically, without a prior. The p value is NEVER enough information.

The argument is where you informally think about your prior, even if you don't use that word explicitly. The argument is where you say that it's implausible that bibles cause cancer, but more plausible that artificial sweeteners cause cancer. It's where you say that it's implausible that songs make you older, but not that the effect is just random. It's where you say that there's so much existing evidence that triples are a good thing, that the fact that this one correlation is negative is not enough to change your mind about that, and there must be some other explanation.

You always, always, have to make that argument. If you disagree, fine. But don't blame me. Blame Bayes' Theorem.

"Statisticians can prove almost anything"

2011-11-21T10:43:00.003-05:00

Sometimes, when you look for statistical significance, you'll find it even if the effect isn't real -- in other words, a false positive. With a 5% significance level, you'll find that one out of 20 times.

However, experimenters don't do just one analysis one time. They'll try a bunch of different variables, and a bunch of different datasets. If they try enough things, they have a much better than 5% chance of coming up with a positive. How much better? Well, there's no real way to tell, since the tests aren't independent (adding one dependent variable to a regression isn't really a whole new regression). But, intuitively: if, by coincidence, your first experiment winds up at (say) p=0.15, it seems like it should be possible to get it down to 0.05 if you try a few things.

That's exactly what Joseph P. Simmons, Leif D. Nelson, and Uri Simonsohn did in a new academic paper (reported on in today's National Post). They wanted to prove the hypothesis that listening to children's music makes you older. (Not makes you *feel* older, but actually makes your date of birth earlier.) Obviously, that hypothesis is false.

Still, the authors managed to find statistical significance. It turned out that subjects who were randomly selected to listen to "When I'm Sixty Four" had an average (adjusted) age of 20.1 years, but those who listened to the children's song "Kalimba" had an adjusted age of 21.5 years. That was significant at p=.04.

How? Well, they gave the subjects three songs to listen to, but only put two in the regression. They asked the subjects 12 questions, but used only one in the regression. And, they kept testing subjects 10 at a time until they got significance, then stopped.

In other words, they tried a large number of permutations, but only reported the one that led to statistical significance.

One thing I found interesting was that one variable -- father's age -- made the biggest difference, dropping the p-value from .33 to .04. That makes sense, because father's age is very much related to subject's age. If you father is 40, you're unlikely to be 35. You could actually make a case that father's age *helps* the logic, not hurts it, even though it was arbitrarily selected because it gave the desired result.

-----

In this case, all the permutations meant that statistical significance was extremely likely. Suppose that, before any regressions, the two groups had about the same age. Then, you start adjusting for things, one at a time. What you're looking for is a significant difference in that one respect. The chances of that are 5%. But, the things the researchers adjusted for are independent: how much they would enjoy eating at a diner, their political orientation, which of four Canadian quarterbacks believed they won an award ... and so on. With ten independent thingies, the chance at least one would be significant is about 0.4.

Add to that the possibility of continuing the experiment until significance was found, and the possibility of combining factors, and you're well over 0.5.

Plus, if the researchers hadn't found significance, they would have kept adjusting the experiment until they did!

-----

The authors make recommendations for how to avoid this problem. They say that researchers should be forced to decide in advance, when to stop collecting data. And they should be forced to list all variables and all conditions, allowing the referees and the readers to see all the "failed" options.

These are all good things. Another thing that I might add is: you have to repeat the *exact same study* with a second dataset. If the result was the result of manipulation, you'll have only a 5% chance of having it stand up to an exact replication. This might create more false negatives, but I think it'd be worth it.

-----

One point I'd add is that this study reinforces my point, last post, that the interpretation of the study is just as important as the regression. For one thing, looking at all the "failed" iterations of the study is necessary to decide how to describe the conclusions. But, mostly, this study shows an extreme example of how you have to use insight to figure out what's going on.

Even if this study wasn't manipulated, the conclusion "listening to children's music makes you older" would be ludicrous. But, the regression doesn't tell you that. Only an intelligent analysis of the problem tells you that.

In this case, it's obvious, and you don't need much insight. In other cases, it's more subtle.

-----

Finally, let me take exception to the headline of the National Post article: "Statisticians can prove almost anything, a new study finds." Boo!

First of all, the Post makes the same mistake I argued against last post: the statistics don't prove anything: the statistics *plus the argument* make the case. Saying "statistics prove a hypothesis" is like saying "subtraction proves socialism works" or "the hammer built the birdhouse."

Second, a psychologist who uses statistics should not be described as a statistician, any more than an insurance salesman should be described as an actuary.

Third, any statistician would tell you, in seconds, that if you allow yourself to try multiple attempts, the .05 goes out the window. It's the sciences that have chosen to ignore that fact.

The true moral of the story, I'd argue, is that the traditional academic standard is wrong -- the standard that once you find statistical significance, you're entitled to conclude your effect is real.

-----

P.S. If Uri Simonsohn's name looks familiar, it might be because he was one of the authors of the "batters hit .463 when gunning for .300" study.

A research study is just a peer-reviewed argument

2011-11-18T09:25:00.014-05:00

To make your case in court, you need two things: first, some evidence; and, second, an argument about what the evidence shows.

The same thing is true in sabermetrics, or any other science. You have your data, and your analysis; that's the evidence. Then you have an argument about what it means.

But, most of the time, the "argument" part gets short shrift. Pick up a typical academic paper, and you'll see that most of the pages are devoted to explaining a regression, and listing the results and the coefficients and the corrections and the tests. Then, the author will just make an unstated assumption about what that means in real life, as if the regression has proven the case all by itself.

That's not right. The regression is important, but it's just the gathering of the evidence. You still have to look at that evidence, and explain what you think it means. You have to make an argument. The regression, by itself, is not an argument. The *interpretation* of the regression is the argument.

For instance: suppose you do a simple regression on exercise and lifespan, and you get the result that every extra mile jogged is associated with an increased lifespan of, say, 10 minutes. What does that mean in practical terms? Probably, the researcher will say that if you want Americans' lifespan to increase by a day, we should consider getting each of them to jog 144 more miles than they would otherwise. That would seem reasonable to most of us.

Suppose, now, another study looks at pro sports, and finds that every year spent as a starting MLB shortstop is associated with an extra $2 million in lifetime earnings. Will the researcher now say that if we want everyone to earn an extra $2 million, we should expand MLB so that everyone in the USA can be a starting shortstop? That would be silly.

Still another researcher does a regression to use triples to predict runs scored. That one finds a negative relationship. Should the study conclude that teams stop trying to hit triples, that it's just hurting them? Again, that would be the wrong conclusion.

All three of these regressions have exactly the same structure. The math is the same, the computer software is the same, the testing for heteroskedasticity is the same ... everything about the regressions themselves is the same. The difference is in the *interpretation* of what the regressions mean. The same interpretation, the same argument, makes sense in the first case, but is obviously ludicrous in the other two cases. And even the third case is very different from the second case.

The regression is just data, just evidence. It's the *interpretation* that's crucial, the argument about what that evidence means.

Why, then, do so many academic papers spend pages and pages on the details of the regression, but only a line or two justifying their conclusions? I don't know for sure, but I'd guess it's because regression looks mathematical and scholarly and intellectual and high-status, while arguments sound subjective and imprecise and unscientific and low-status.

Nonetheless, I think the academic world has it backwards. Regressions are easy -- shove some numbers into a computer and see what comes out. Interpretations -- especially correct interpretations -- are the hard part.

-----

If you think my examples are silly because they're too obvious, here's a real-life example that's more subtle: the relationship between salary and wins in baseball, a topic that's been discussed quite a bit over the last few years. If you do a regression on 2009 data, you'll get that

-- the correlation coefficient is .48
-- the r-squared = .23
-- the value of the coefficient is .16 of a win per $1 million spent
-- the coefficient is statistically significant (as compared to the null hypothesis of zero).

That's all evidence. But, evidence of what? So far, it's just numbers. What do they actually *mean*, in terms of actual knowledge about baseball?

To get from the raw numbers to a conclusion, you have to interpret what the regression says. You have to make an argument. You have to use logic and reason.

So you look the coefficient of .16. From that, you can say, in 2009, every extra $6 million spent resulted, on average, in one extra win. I'm happy calling that a "fact" -- it's exactly what the data shows. But, almost anywhere you go from there now becomes interpretation. What does that *mean*, that every extra $6 million resulted in an extra win? What are the practical implications?

For instance, suppose you're a GM and want to gain an extra win next year. How much extra money do you have to spend on free agents? If you want to convince me that you know the answer, you have to take the evidence provided by the regression, and *make an argument* for why you're right.

A naive interpretation might be to just use that $6 million figure, and say, that's it! Spend an extra $6 million, and get an extra win. It seems obvious from the regression, but it would be wrong.

Why is it wrong? It's wrong because there are other causes of winning than spending money on free agents. There's also spending money on "slaves," and spending money on "arbs". Those are much cheaper than free agents. Effectively, some teams get wins almost for free, by having good young players. The teams that don't have that have to spend double, as it were: they have to buy a free agent just to catch up to the team with the cheap guys, and then they have to buy another one to surpass him.

For instance, team A has 80 wins for "free". Team B has 70 wins for "free" and buys another 20 on the free-agent market. The regression doesn't know free from not free. It sees that team B has 10 more wins, but spent an extra $20X dollars, where X is the actual cost of a free agent per win. Therefore, it spits out that it took 2X dollars to buy each extra win, even though it only took X.

That is: the coefficient of dollars per win from the regression is twice what it actually costs to buy one. The coefficient doesn't measure what a naive researcher might think it does.

My numbers are artificial, but I chose numbers that actually come fairly close to real life. Various sabermetric studies have shown that a free agent win actually costs $4.5 million. But regressions for 2008, 2009, and 2010 respectively show figures of $8.9, $6.2, and $12.6 million, respectively -- about twice as much.

Again, the issue is interpretation. If you're just showing the regression results, and saying, "here, figure out what this means," then, fine. But if your paper has a section called "discussion," or "conclusions," that means you're interpreting the results. And that's the part where it's easy to go wrong, and where you have to be careful.

----

Which brings me, finally, to the point that I'm trying to make: we should stop treating academic studies as objective scientific findings, and start treating them as arguments. Sure, we can remember that academic papers are written by experts, and peer reviewed, and that much of the time, there's no political slant behind them. If we want, we can consider them as generally well-reasoned arguments by experts of presumably above-average judgment.

But they're still arguments.

So when an interesting study is published, and the media report on it, they should treat it as an argument. And we should hold it to the same standards of skepticism to which we hold other arguments. A research paper is like an extended op-ed. The fact that there's math, and a review process, doesn't make them any less argument-like. The New York Times wouldn't present Paul Krugman's column as fact just because he used regressions and peer review, would they?

I googled the phrase "a new study shows." I got 55 million results. "A new study claims" gives only 4 million. "A new study argues" gives only 300,000.

But, really, It should be the other way around. New studies normally don't "show" anything but the regression results. Their conclusions are always "claimed" or "argued".

-----

The word "show" should be used only when the writer wants to indicate that the claim is true, or that it has been widely accepted in the field. At the time his original Baseball Abstract came out, you'd have to say Bill James was "arguing" that the Pythagorean Projection is a good estimator of team wins. But now that we know it's right, we say he "showed" it.

"Show" implies that you accept the conclusion. "Argue" or "claim" implies that you're not making a judgment.

The interesting thing is that the media seem to understand this. Sure, 90 percent of the time, they say "show". But when they don't, it's for a reason. The "claims" and "argues" are saved for controversial or frivolous cases, ones that the reporter doesn't want to imply are true. For instance, "New study claims gun-control laws have no effect on Canadian murder rate." And, "a new study argues that poker is a game of skill, not chance."

It's as if the reporters want to pretend scientific papers are always right, unless they conclude something that the reporter or editor doesn't agree with. But it's not the reporter's job to be implying the correctness of a conclusion, unless the reporter has analyzed the paper, and is writing the article as an opinion piece.

Ninety-nine percent of the time, a research paper does not "show" anything -- it only argues it. Because, correct conclusions don't just pop out of a regression. They only show up when you support that regression with a good, solid argument.

The main economic benefit of baseball: we love it

2011-11-10T14:53:00.004-05:00

From "The Sports Economist," here's commenter "bobby" with a very good comment:

I find it mildly distressing that almost all of the discussion about economic impacts of sporting events is about rectangles with rarely if ever a discussion of triangles. I was always trained that welfare was measured by consumer and producer surplus, not expenditures, but then what do I know?

...

I guess the idea that people are happier with a baseball game than a movie doesn’t mean much anymore, and its downright silly to suggest that a baseball game makes a place better off because people could have gone to a movie instead.

What he's saying is that if you're trying to measure the benefit of something, you measure it by "consumer surplus." That's the economic term for the difference between what you have to pay for something, and the maximum price you'd be willing to pay for it.

A lot of things have a huge consumer surplus. Take, for instance, a headache pill. An ibuprofen tablet costs only a few pennies -- a dime, tops. How much would you be willing to pay to make your headache go away? It's at least a dollar -- probably more, but at least a dollar. That means that every time you buy an Advil for ten cents, you're making a "profit" of at least 90 cents.

An easier way of looking at it is this: if the product you're using didn't exist, how much worse off would you be? That's consumer surplus.

There's consumer surplus in almost everything you pay for. That's because, if you didn't buy it, you'd have to buy something else you liked less. If there were no Tim Hortons, I'd have to buy Starbucks coffee, which I don't like as much. If my favorite restaurant closed, I'd have to go to my second favorite, which is still good, but not as good as my favorite. And so on.

Now, for sports: how much consumer surplus do you get from sports? How much worse off would you be if there were no baseball, or hockey? For most of you reading this, your answer is probably -- a lot worse. You'd have more money, because you wouldn't be spending on trading cards and tickets and Bill James Baseball Abstracts, but, that barely matters, compared to how much less interesting your life would be without baseball. The same is probably true for your favorite team, if you have one. My life would be a lot worse without the Toronto Maple Leafs, even if all the other teams were still around.

So Bobby's argument is quite correct. The most important economic consideration, when it comes to pro sports is how much better off people are because of it. Why, then, in almost every discussion of sports and economics, is this not a consideration?

One reason, as Victor Matheson says here, is that economists are often reacting to questions from non-economists, or politicians, who are more concerned about GDP and job creation than about the intrinsic value of sports as entertainment. When a candidate for office talks about building a new stadium to attract a team, the economic arguments are about the monetary values of the transactions it will create, rather than how happy the fans will be. And so, that's what the economists have to respond to.

Most of the time, though, the answer is that a new sports team creates only negligible zero jobs on net, and little increase in GDP. Because, after all, if the money didn't get spent on sports, it would get spent on something else. If you live in Montreal and don't have the Expos any more, you'll go to a movie, or go out to dinner instead.

That's true for almost anything. If medicines were made illegal, GDP wouldn't change much in the long term -- you'll take the dime you would have spent on an Advil, and spend it on something else instead. The main reason a headache pill is a good thing is not that it adds 10 cents to total output, but that its benefit is way, way higher than its cost.

What I'd like to see, in economic analysis of sports, is some kind of estimate of how much it improves people's lives. It's a lot. Matheson says in his post that there have been some attempts to quantify consumer surplus, but his example is only among people who pay for tickets. But what about everything else? Most of us benefit from baseball far, far more than just our ticket purchases. We watch games on TV, write blogs about them, analyze them, talk about them at the water cooler. Sports are a big part of the fabric of most of our lives, and having a team in our city to root for is a huge unmeasured happy benefit.

I have argued before that if it makes sense for government to subsidize things like public broadcasting (CBC, BBC, NPR, etc.), it should also make sense for it to subsidize a hockey team, on a cost/benefit basis. But I can't show you any evidence (other than back-of-the-envelope) that that's the case, unless and until the economists start listening to bobby, and get to work on showing the size of the benefit.

"Baseball Analyst" archives now available

2011-11-02T15:35:00.005-04:00

In 1982, Bill James created the "Baseball Analyst," a bimonthly amateur sabermetrics journal that relied on contributions from readers. It ran 40 issues, dying in early 1989.

Last weekend, with Bill's permission, I scanned all my issues and sent them to Jacob Pomrenke of SABR. Stephen Roney contributed some pages I was missing. Jacob reformatted everything. Rob Neyer, who was responsible for the Analyst's last few issues, wrote an introduction.

Finally, Jacob put it all online at the SABR website. All 40 issues are now publicly available for download.

Great stuff ... thanks to Jacob, Stephen, Rob, and especially Bill!

A rudimentary NFL season simulation

2011-10-31T14:56:00.003-04:00

Following a post by Tango a couple of weeks ago on the playoff systems of the various sports, I thought I'd try writing a simulation. This is an update of that work in progress. Actually, I've only started on the NFL, and I haven't even done playoffs yet, just the regular season. But I thought I'd at least share what I've got so far.

In the simulation, each of the 32 teams was assigned a "true talent," from a normal distribution with mean .500 and standard deviation .143. No team was allowed to have talent higher than .900 or lower than .100; if they did, they were moved to .900 or .100. Then, all 32 teams were moved the same amount (arithmetically) in the same direction to get the overall talent to average exactly .500. (I think this method actually reduces the expected SD below .143, but I didn't bother fixing that.)

The 16-game schedule is random, instead of unbalanced (with the restriction that a team can't face any another team more than twice). There are no tie games. There is no home field advantage (although that would be easy to add in). The chance of winning each game is determined by the log5 method. There are no ties in games. Ties in the standings (division or wild card) are broken randomly.

As I said, I stopped there for now; haven't done playoffs yet. That's the next step, along with home field advantage.

Anyway, here are some results. Each result is out of 100,000 seasons. Every result came from a different run of the simulation. Results varied a fair bit per run, but I think everything is reasonably typical.

--------

I checked for all teams out of 3,200,000 (32 teams, 100,000 seasons) that finished more than 8 games above or below their talent. That's hard to do, obviously. Also, the worse or better you are, the harder it is. It's (relatively) easier for an 8-8 team to go 16-0 than for a 3-13 team to go 11-5. Amplifying that is the fact that there are a lot more 8-8 teams than 3-13 teams. However, offsetting that, a little bit, is the fact that the 3-13 team can also go 12-4 or 13-3 or better.

In any case ... there were 43 cases where a team differed from its talent by 8 games or more. Of those, 26 were teams that outperformed, and 17 were teams that underperformed.

The biggest differential was in season 98,534, where the Broncos a team that had talent of 4.56 wins (out of 16), but went 14-2, for a differential of 9.44 games. That was the only team with a differential of 9 or more. Part of the reason it did so well was that it faced inferior opponents. You'd expect any given team's opponents to average 8.00 games of talent. But in that season, the Broncos' opponents' talent was only 7.45 games. Not a huge difference, but still.

Actually, when it comes to extreme events, a small difference in opponents makes a big difference in probability. Of the 43 teams in the sample, 38 of them had records that went in the direction "aided" by the opposition (in the sense that the underperforming teams played better-than-expected opponents, and vice versa). That's 38-5 in favor.

The worst team in the sample was the season 63,924 Jets, a 3.03 team that went 12-4 (playing 7.11-win opponents). The best team in the sample was a Bucs team that was expected to win 12.04 games, but instead went 4-12 (playing 8.58-win opponents).

--------

I also took a look at teams that went 0-16.

Those results probably aren't as realistic, because they're heavily dependent on the shape of the tail of the talent distribution ... and we really don't know what that is. Recall that we chose a normal distribution that gets truncated at .100 (1.6 wins). Both those choices -- normal, and truncated -- are arbitrary and probably not close enough to real life. (Also, teams could drop below .100 in talent from the adjustment that sets all league-years to .500.)

In addition, the other shortcuts in the simulation probably skew the results too. The mainstream results are probably right, but the extremes are extremely sensitive to some of the assumptions.

With those caveats: there were 5,663 of those 0-16 teams out of 3.2 million, and their average talent was .181, which is just under 3-13. I suspect the talent of actual flesh-and-blood 0-16 teams is higher than that, but I really don't know.

16-0 should be exactly symmetrical, so I won't show that separately.

--------

I checked for four-way ties where every team has the same record. That happened 878 times out of 800,000, or about once every century.

There were five seasons out of 100,000 where two divisions had a four-way tie. Actually, that might be a little high ... the test runs had only 1 or 2 such seasons.

---------

Anyway, before I start on the playoffs, and repeating this for other sports leagues, I'm looking for feedback on what I've got so far. Any suggestions?

And, if you want me to run the sim and check for something in particular, let me know in the comments. It's real easy to add a couple of lines of code to check for something specific.

Being proven wrong is like winning the lottery

2011-10-26T18:39:00.010-04:00

Following my last post on being a dick in online message boards, I was going to prepare a list of things that can really piss off people who are trying to have a discussion with you, things that you shouldn't do. Like, for instance, changing the subject when someone brings up key points, or being inappropriately picky in ways that don't affect the main point. That kind of thing.

As I was preparing that list, it occurred to me: these are things that nobody would ever do who really, truly cared about whether they were right or not. They're things that people do when they're trapped by logic, but they're reluctant to admit their position might be wrong.

Why don't people want to admit they were wrong, even when it seems obvious to everyone else? Ego. Nobody likes to be proven wrong. But, if we can get over that pride thing, we should WANT to be proven wrong! It makes things better for us.

Because, it's bad to be wrong. Therefore, if you're wrong, the best thing is to *stop being wrong*. And the way to stop being wrong is to change your mind.

Also, there are lots of side benefits to changing from a wrong view to a correct view. For one thing, you're suddenly more right! And, for another, now being right lets you see things in a whole new way.

If you think that 1+1 equals 3, the world isn't going to make a whole lot of sense to you. You'll think your paycheck doesn't add up, and everyone's giving you the wrong change. You'll think the IRS is ripping you off, and mathematics is just arbitrary, since everyone just seems to be totaling things a different way. You'll be suspicious of everyone, and life is going to be pretty difficult.

But, when you see that 1+1=2, suddenly, everything comes into focus! You'll see the world works logically, after all. You now predict what people will do, and see that everything really does add up. Your life is better -- a lot better.

So, you should be happy and excited at the idea that we might be wrong. The guy you're getting mad at could be the one that changes your life, if it turns out that he's right.

--------

Another way to look at it: suppose you're a successful major-league pitcher, making a few million dollars a year. You go to the doctor, and he says, hey, there's been something wrong with your elbow since birth.

Should you be insulted and angry that the doctor is insulting your awesome pitching arm? No, you should be very excited -- that's great news! If the doctor is right, and he the problem, maybe you'll be throwing 100 mph instead of 95!

Don't like that analogy? OK, here's another try.

Let's suppose you open a restaurant, and you're very successful, and people like your food. You're very proud of being a great chef. Then, someone tells you, correctly, that one of your appetizers, one that you think is one of your best, is actually pretty awful. Your customers hate it.

Your first reaction might be to get defensive. But, again, you should be thrilled! Now you can fix that dish. Your food, your restaurant, your profit, and your reputation will all be better than before. It's almost the best thing that can happen to you. ~~Being~~ Finding out you're wrong is like winning the lottery!

------

So, in online debates, we should all argue as if God will give us a million dollars if we're proven wrong. That way, when someone disagrees with us, we won't just dismiss him out of hand or twist our logic to try to save our own position. We'll stop and think, "hey, is it possible that this guy has my million dollars?" We'll be less likely to let our ego take over and turn us into dicks.

Okay, maybe that's too much money. Maybe for a shot at a million dollars, we'd listen to idiots way, way too long. Maybe the right amount is, I dunno, a hundred dollars, or something. But you get the idea.

And I should add that I'm not saying that I, personally, know how to suppress my own ego, or even that I succeed in doing it when I try. I'm just saying that I know I *should*.

-----

I guess my overall point is that any online discussion, even between people who violently disagree with each other, should be a co-operative venture. One of you is wrong, and you're working together to find out who. And, we should keep in mind that most of the benefit goes to the person who was actually wrong in the first place.

When someone you respect, or someone who seems to be expert and knowledgeable, starts disagreeing with you, it's like you've stumbled upon a fistful of lottery tickets. Argue your position, yes, but don't get defensive, and keep an open mind. Sure, it might be that other guy who's wrong. But if you're really, really lucky, it'll be you.

In defense of online rudeness

2011-10-22T12:08:00.005-04:00

A couple of weeks ago, the moderator of a certain website I frequent posted a message, reminding us commenters to respect each other. The warning wasn't random, of course; it was prompted by a discussion that got a little less civilized than normal for that site.

This kind of thing plays out all the time on thousands of different sites. But what bugs me about these "please be civil and respect each other" warnings is that they only target one general type of rudeness. I did a Google search for "message board rules etiquette," to get some examples of posted rules. A lot of them include something like this:

"Don't issue personal attacks, use profanity, or post threatening, abusive, harassing, or otherwise offensive language or images. Keep your messages appropriate and courteous at all times. Please disagree with other opinions respectfully."

Now, most of us follow this advice almost all the time. But, sometimes, we don't. When don't we? When we get really frustrated with someone. Why do we get frustrated?

Well, they might be repeatedly misrepresenting something we wrote. They might be ignoring our questions. They might be following the argument back and forth for hours, until they realize they're "losing", and change the subject. They might be obviously disingenuous, denying something they wrote in another post just days before. They might agree with what you say when it suits their argument, but change their mind as soon as it goes against them. They might be trolling for fun. They might be committing any one of a thousand logical fallacies, and refusing to be corrected.

But ... the rules don't prohibit that, do they? The rules say you can get kicked off the forum for calling someone an idiot. But you can't get kicked off for repeatedly (and perhaps deliberately) butchering your logic.

Here's a hypothetical situation I made up:

A: US citizens spend too much on foreign aid. We need to help our own instead. There are thousands out of work. I don't know why you don't see that.

B: I see that, but money spent outside the US can help a lot more people who are desperate. We can save hundreds of lives for almost nothing.

A: I don't believe you.

B: (Goes and searches the internet. Writes several paragraphs of illustration of various public health costs, and how cheap it is to save lives in Africa with cheap drugs or vitamins or something.)

A: Well, maybe, but the multinationals make too much profit when we do that.

B: (Goes and searches the internet) Here are some companies selling drugs at cost, or offering them for free, if we just pay to distribute them!

A: Yeah, whatever. And, regardless, we still spend too much. Charity is nice, but in moderation. We should spend only about half of 1% of our income on foreign aid. That's $1 out of every $200. That's my maximum.

B: Hang on, let me search the internet ... well, as it turns out, we spend only 0.1% of our income on foreign aid! So, we both agree that we could spend a little more, right?

A: I said we should spend half of that! See, we spend too much!

B: No, half of that would be 0.05%. You said 0.5%. Here, let me quote your post: "$1 out of every $200".

A: You're misquoting me. Besides, you can throw numbers at me all you want. They're just numbers. The fact is that we spend too much on foreign aid. We need to help our own instead. I don't know why you don't see that.

B: You're a dick.

What happens next? B gets in trouble for calling A a dick. But A gets off scot-free for BEING a dick. And that's a lot worse.

As far as I'm concerned, it should be A who gets kicked off the site, not B. But that'll never happen. See, B's offense was objective, and easily caught. He used the word "dick". Everyone can understand that, and you can easily make a rule out of it. "Why did B get kicked off?" "He called A a dick." "Oh, OK."

On the other hand, A's offense was subjective. It requires a judgment call from the moderator. And, there's no smoking gun. "Why did A get kicked off?" B would say, "He was a dick." But A's supporters would say, "It's because he didn't agree with B." "It's because the moderator didn't like his politics." "It's because nobody respects B's right to stand up for American workers." "It's because that website doesn't respect dissenting views."

It's hard to describe what A did wrong. The evidence is hard to describe. There's no smoking dick.

And so, the As of the world get away with it, and we just have to put up with them.

--------

I don't think that that B said anything morally objectionable. A indeed WAS a dick. Sure, it didn't *have* to be said ... but, it was true. At least in the sense that "you're a dick" can ever be said to be true.

And I would argue that B, after investing so much time and effort in moving the argument forward, earned the right to say it.

Sure, you don't want discussions degenerating into name-calling. That's no fun for anyone. But, in the appropriate context, an occasional, controlled outburst is OK.

Strained analogy: think of a discussion as a pot luck, and insults like ketchup. If you show up at a pot luck, you don't just bring a bottle of ketchup. That's rude, and tacky. But, if you bring hamburgers for everyone, and you *also* bring a bottle of ketchup ... that's perfectly OK. In fact, the hamburgers you're providing are actually enhanced by the ketchup you brought.

The rule is, if you want the right to serve ketchup at the pot luck, you have the obligation to serve the meat to go with it. And, if B has just spent the better part of an hour researching foreign aid, and hundreds of characters typing rebuttals to a poster he thought was arguing in good faith ... that's a LOT of meat. You've got to say that B has earned the right to pound the ketchup bottle a little bit.

---------

This is one of the problems we have with public discussion in general. Whether it's politicians, columnists, academics, or talking heads on TV, the unwritten code is the same. You can butcher logic all you want, and nobody will call you on it. But resort to name-calling, or other "uneducated" forms of language, and you get in trouble.

The public doesn't have the time or patience to judge what you've said. But it does understand *how* you said it!

You probably know what happened to Don Cherry a couple of weeks ago.

Over the summer, in separate incidents, three former NHL players took their own lives. They had all been "enforcers," players kept on the team for their willingness to start fights with opposing players. Speculation ensued that there was somehow a link between being an enforcer and having mental health issues. Three other former enforcers, Stu Grimson, Chris Nilan, and Jim Thomson, apparently made comments that expressed support for that hypothesis.

Don Cherry went on TV and accused Grimson, Nilan and Thomson of hypocrisy. Those guys, themselves, once chose to make a very good living with their fists, Cherry said, knowing that they'd be out of professional sports if they didn't. Now, they're trying to deny the same choice to today's players, since, with their careers long over, banning fighting would not longer cost them anything. And this, Cherry argued, was on the basis of a flimsy hypothesis with no solid evidence behind it. Hypocrites!

Seems like a legitimate argument, right? I mean, you can certainly disagree with it, but it's not that unreasonable a point to make, in the context of a controversial issue that's already attracted a lot of discussion.

But Cherry got himself in trouble. Why? Because he didn't use educated language. He didn't say it formally, the way I described it. He used less fancy words. One word, in particular: "pukes." Cherry called Grimson, Nilan, and Thomson "a bunch of pukes."

Without the word "pukes," it's just another Don Cherry TV segment. But with the word "pukes," suddenly there's outrage. There were news stories, outraged columnists, and even newspaper editorials, all of them prominently featuring the word "pukes."

In the midst of all this, Grimson threatened to sue Cherry if he didn't apologize. That was more than an idle threat: Grimson is a lawyer, and his statement threatening "further recourse" was issued by his own law firm. Cherry apologized a few days later. Grimson issued a new statement that said, OK, he wouldn't sue, but maybe the CBC should fire Cherry anyway.

Cherry got a raw deal: not just because of the content, which wasn't really any more controversial than his usual, but because of the attention he got by breaking the taboo against name-calling. "If you call somebody a name," goes the unwritten, unspoken rule, "it signals that you're uneducated and boorish, so, accordingly, we will oppose your argument exceedingly vigorously."

The media and public would do better to take some now-famous advice from MGL: "If you guys can’t separate tone from substance, that is your problem not mine. Stop being such whiners about tone.”

--------

P.S. Kind of off topic, but while I'm here ...

I don't agree with Cherry's logic that Grimson is a "puke" for speculating on the link between fighting and mental health. I think that's a perfectly reasonable thing for a former goon to wonder about.

But, I do have a problem with Stu Grimson's conduct afterwards.

I mean ... What kind of guy spends his career beating people up, then complains about a "lack of decency" because someone has the temerity to criticize his views about it? What kind of guy threatens to sue someone just because he's been called a childish name? What kind of guy speaks out on a position of public importance, and then when someone disagrees, tries to get him fired? What kind of guy would leverage his advantage -- knowing he's a lawyer and can cause all kinds of problems for Don Cherry at no cost to himself -- to extort an apology with credible threats of a lawsuit?

Maybe Don Cherry was right about Grimson, after all.

How much does "Moneyball" help a team?

2011-10-19T10:43:00.005-04:00

How much is sabermetrics worth to a team?

That's probably a hard question to answer. Every team uses statistics to some extent. Even before sabermetrics, teams were looking at player statistics to decide who to play and who not to play. They may not have had any fancy formulas, but they had a pretty good idea of how to weight the relative contributions of players. Nobody ever released a 30-HR guy because he was only hitting .240, and nobody ever released a .330 hitter because he had no power. Intuitive evaluations weren't perfect, of course, but they were pretty reasonable most of the time.

Where sabermetrics helps, I think, is not in evaluating actual performance, but in helping figure out *future* performance. How to extrapolate minor-league performance in to major league performance ... how to take luck out of a player's batting or pitching line ... figuring how different kinds of players age ... that sort of thing.

Suppose you took a team management right out of the early 1970s, and gave them a team today without letting them learn anything discovered after 1977. How much would that team underperform compared to the rest of MLB? I don't have an answer to the question, but I'd be interested in hearing yours.

Anyway, here's a narrower question. How much can a more sabermetric approach *today* benefit a team, compared to, say, the typical team's sabermetric approach? For instance, how much did Billy Beane really mean to the A's?

A couple of weeks ago, Tango did a study to figure out which teams did better or worse than expected, given their payroll. The A's were the team that outperformed the most over the last decade -- about 7 games per season, it looks like. That's a lot, but there's probably a whole bunch of luck there, since we're cherry-picking them as the best of the lot. Also, it's possible that much of their outperformance came in the early years, when, as many critics of "Moneyball" hype have pointed out, they had three underpriced ace starters.

So, we'd have to regress that 7 games to the mean a fair bit. If you made me make an arbitrary guess, I'd be willing to bet that less than half of that seven game advantage came from sabermetrics. (But, I have no real basis for that guess without studying it.)

Anyway, with the Cubs signing Theo Epstein, we now have a market estimate for what sabermetrics might be worth today. Epstein's new agreement is for about $4 million per season. He still had one year to go on his contract with the Red Sox, for which they will receive some sort of compensation from the Cubs. Let's say that compensation will be worth $1 million. So Epstein's value is around $5 million. I don't know how much an ~~average~~ replacement level GM makes, by comparison. To be conservative, let's say it's $500,000, although it's probably more than that. That means that Epstein's excess value is $4.5 million, exactly what it costs in free agent players to gain one extra win.

It looks like that's what Epstein is worth: one win per season.

Is that a lot? Frankly, I don't know. It's a competitive market for players these days, with lots of money on the line, and there's lots of random luck in who makes it and who doesn't. In that light, it could be that one win per season is an exceptional, genius-level performance.

If that's the case, doesn't it mean that the "Moneyball" approach is overrated? I mean, one win a year. At that rate, it would take decades, even centuries, to have good statistical evidence that the sabermetric approach works.

Of course, you have to remember that that's compared to other teams ... and, nowadays, those other teams are doing a fair amount of statistical work themselves. Maybe it's three or four games over a team that won't look at anything new at all, that never heard of Voros McCracken and winds up overpaying pitchers with lucky BABIPs. And, maybe Epstein took less pay than he was worth in order to become a Cub. Maybe it's a win and a quarter, or a win and a half.

Still ... to me, one game doesn't seem that unreasonable. The point might not be that an you can win pennants just by embracing sabermetrics. The point might be that, with every team in a sabermetric arms race against every other team, you certainly can *lose* pennants if you persist in living in the 70s.

But, again ... one game. Doesn't that mean that if a team does well, and someone credits "Moneyball," they're probably just blowing smoke?

UPDATES:

1. In the comments, Bill Waite suggests that sabermetrically-savvy managers might have a significant impact, too. He says that just rejigging the lineup is worth almost half a game a season, and says that the difference between best and worst could be as much as eight games.

Food for thought. It would be interesting to consider how to try to look for this in the historical record (if indeed that is possible), since we know that some managers are indeed more numbers-oriented than others.

2. Matt Swartz e-mailed me about a study where he found a positive correlation between sabermetric management and team performance. It's here.

Capital Gains and Warren Buffett: Part II

2011-10-15T15:04:00.004-04:00

This is a continuation of the previous post about why capital gains taxes should be lower than regular taxes, and perhaps even zero.

-------

3. More Double Taxation

One year, the New York Yankees draft Mickey, a player with star potential.

The team signs him to a five-year contract at $2 million a year. They figure that's a bargain. Mickey is good enough that he's expected to produce $15 million a year in revenue, so the Yankees figure, reasonably, that they're making $65 million on the deal.

The day after the draft, along comes the IRS. "Hey," they say to the Yankees. "You just signed a deal that's worth $65 million in profit. We want the corporate tax on that, $22 million. Pay up!"

The Yankees say, "Wait a second! Mickey hasn't played a single game for us yet, so we haven't made any profit! How can you be asking for the tax already?"

The IRS says, "We don't want to wait. We want the money now. But, don't worry, we know we're collecting the money in advance, so we're discounting the amount owed by the current interest rate."

The Yankees protest. "That's still not fair," they say. "You shouldn't tax us until we actually make the money. After all, Mickey just signed a contract for $10 million. You're not taxing *him* on that $10 million right away ... you're going to tax him only when he gets his paychecks. You're not going to his house and demanding $4 million from him right this second, are you?"

The government agent replies, "I don't know why you have a problem with this. I already told you we're adjusting for the time value of money. And we know you *have* enough cash to pay us now, because you're the Yankees. So why not?"

"What if Mickey doesn't perform and we don't make the expected profit?" the Yankees ask.

"No problem," replies the IRS. "We'll give you a partial refund. But, of course, if Mickey starts playing like Albert Pujols, and you clean up, we'll ask for more."

"Hmmm," the Yankees owner says. "I guess it doesn't really matter to the long-term bottom line. But it still seems like it's not right."

--------

Is it unfair? Yeah, I think it is ... if the Yankees haven't made any profit yet, they shouldn't pay any tax on that profit.

Still, the present value of the tax paid is the same, whether the Yankees pay it in a lump sum, or whether they pay it on a year-by-year basis. So if paying in advance is unfair, it's unfair only from a timing standpoint. Either way, the Yankees pay the right amount of tax. It's just that one way, the IRS is in an ungodly hurry, that they have no business being in. But it's the same amount of money. You might even imagine a situation where the Yankees would *choose* to pay in advance.

So this is just slightly unfair. What would make it *really* unfair is if the IRS made the Yankees pay BOTH WAYS. First, if they made them pay $22 million on their projected $65 million profit, and then they also made them pay regular corporate tax on the annual $13 million profit. That would be pure double taxation, right?

It has to be one or the other. Either they pay the tax on the projected profit in advance, or they pay the tax as the profit comes in. If you make the Yankees pay both, you really are making them pay exactly twice as much tax. Instead of 35% tax, they'd be paying 70%.

Now, if you don't stop to think about it at all, it might not seem unfair that way. When they sign the contract, the accountants say, "Congratulations, Mr. Cashman, that deal you just made with Mickey is worth $65 million." And, at the end of the next five years, the accountants say, "Congratulations, Mr. Cashman, you made $13 million more this year because of Mickey." It may be tempting to add all those up to get $130 million in profit. But you can't do that. The $65 million is THE SAME MONEY as the five $13 millions.

Tax the profit in advance, or tax it when it comes in. Choose only one. If you do both, it's grossly unfair.

With me so far?

--------

Now I'm going to argue that when you have a capital gain on a stock, this is almost exactly what happens: the profit gets taxed twice: once in advance, and once when it comes in.

You buy a stock in a drug company, Acme, for $1 a share. Acme is just barely breaking even. After you buy the stock, Acme discovers a new drug that prevents heart attacks. It's going to be a blockbuster. In fact, it's so good that it's going to produce annual earnings and dividends of $2 a share, indefinitely. But, first, the FDA has to approve the drug. It's so good, with no side effects, that FDA approval is assured, but it will still take two years.

Wall Street immediately calculates that $2 a share, indefinitely, starting in two years, makes the stock worth $31. Shareholders celebrate. You sell your share of Acme for $31, incurring a $30 capital gain. Should you pay tax on that capital gain?

Look what's happened. Acme hasn't actually made any profit yet. The $30 increase in the stock price represents *future earnings*, just like the $65 million contract with Mickey is future earnings. So if you tax the $30 increase -- which is what capital gains tax does -- and then you later tax the corporate profits -- which of course you will do -- that's exactly analogous to what happens if you tax the Yankees twice.

To make things fair, the IRS should pick one: either tax the capital gain right now, or tax the corporate profits later. One or the other.

Otherwise, the IRS is getting a lot more than the corporate tax on the profits. Because, the fact that I sold my share does not change the future tax bill of the corporation, or the new shareholder, at all. Whether I sell the share or not, the corporation will still pay the normal taxes on the profits. But, by me selling the share, I trigger an *extra* tax, a capital gains tax, on the value of the future profits. That's two taxes on the same money.

The only thing that makes this example different from the Yankees example is that the two taxes are paid by two separate people, instead of one. That kind of hides the unfairness. The original buyer pays the full, future corporate tax, which seems OK because he just made a big gain on his stock. And the new buyer pays the full, future corporate tax, which seems fair because he's the owner at the time the profit was made.

But, still, they're both paying tax on the *same money*. [See footnote 1.]

--------

As I said, it should be one or the other. Since it's hard to discount the corporate tax by capital gains tax that's already been paid, the easiest solution is to just eliminate the capital gains tax. That way, the profit is only be taxed once, when the corporation earns it.

That may seem unfair, because, after all, when you earned your $30 capital gain on your $1 investment, that's a lot of money. Why shouldn't you pay tax on it?

One answer is: you already did, kind of.

When you sold the share at $31, that was the market price. That market price is based on after-tax profits. We said the corporation pays a $2 dividend. That means it will make $3 in profits and pay $1 in corporate tax, leaving a $2 dividend. The buyer of your share gets the $2 dividend, and pays 15% tax on it, leaving $1.70 in his pocket.

That $1.70 in his pocket, year after year, is why the new buyer was willing to pay $31 for your share. That means he's looking for an after-tax return of 5.48% on his money, which is what he's getting: $1.70 divided by $31.

Now, suppose there were no corporate tax or dividend tax. Then, instead of $1.70 a year, the new owner of the share would receive the full $3 profit. For the same yield of 5.48% after tax, he'd now be willing to pay $54.74. So, if there were no taxes, the shares would have been worth $54.74 instead of $31, and you would have made a capital gain of $53.74 instead of $30.

So the corporate tax actually did cost you money! It cost you almost exactly the same rate as the overall taxes on the profits -- in this case, about 45% -- because you received only $31 instead of $54.74. That is, in order to take over *your* obligation to pay all that future tax to the government, the new buyer demanded a discount of $23.74 to compensate him for taking over your obligation.

It may look like you're not paying any tax on your capital gain, but, you are, in a hidden kind of way.

Look at it this way. Suppose that while you owned Acme, the government decided to double the corporate tax on drug companies, forever into the future. What would happen? Well, because of the new tax, Acme would only be able to pay $1 in dividends, instead of $2. That means the new owner would receive only 85 cents a year.

But the new owner still wants his 5.48% after-tax yield. So, he's not going to pay $31 for the stock any more. He's only going to be willing to pay $15.50. So that's what you'll have to sell it to him for.

See? The extra corporate tax comes right out of your pocket, even if you're not the one writing the check. [See footnote 2.]

----------

Here's another way to look at it: when you sell the company, you sell it for $54.74, its full value if there were no taxes, but you promise the buyer to pay all the taxes into the future. Instead of paying every year, though, you hand the buyer a cheque for $23.74, and say, "here, invest this and it'll pay your tax bill every year." The buyer says, "OK, that's a deal."

Instead of you actually physically paying the $23.74, you just take it off the price of the share, which is why you wind up with $31. It doesn't look like you paid any tax, but you actually paid all of the corporation's tax into the future!

And, again, that's WITHOUT a separate capital gains tax. With a full capital gains tax, you'd be paying double.

---------

Here's a simpler case.

You grow apples. There's a 50% income tax on the apples you grow. There's also a 50% capital gains tax.

You take two apple seeds, plant them, nurture them, fertilize them, and grow them into two healthy trees. The apples start growing. You harvest them and send half the apples as your income tax. The government, therefore, gets one tree's worth of apples a year -- half the harvest of the two trees.

Now, you sell the two trees to your neighbor. Since you grew them from nothing, your capital gain is the entire two trees. So, at a 50% capital gains tax, you owe the government one tree. But you're out of trees, so you have to buy one tree back from your neighbor, which you now give to the government.

What's happened? The government used to get one tree's worth of apples a year. Now it gets 1.5 trees' worth! First, the tree that you just paid it in capital gains tax, which now belongs 100% to the government. Second, half the apples from the other tree the neighbor now owns.

So, effectively, the government is now taxing the apples not at 50%, but at 75%.

But it's hidden between the two of you. You pay the first 50%, and your neighbor pays 50% of what's left.

--------

It may seem counterintuitive that Warren Buffett should get away with almost no tax on his capital gains. You have to look hard to see that even if the capital gains tax rate is zero, Warren Buffett, and the rest of us, are still, in a hidden way, paying the normal rate.

It may not be obvious, but I think it's true.

--------
Footnote 1: This argument applies when the profit goes on indefinitely. If not, the double taxation isn't quite as bad. Suppose the patent on the drug runs out in 20 years. At that point, the stock will tank and the new buyer will have a capital loss, exactly offsetting the capital gain the old buyer had 20 years before.

However, the government will have earned interest on that capital gains tax for 20 years, before refunding it without interest. After 20 years, because of forgone interest, the refund might be worth, say, only 1/4 the original amount. So it's not double taxation, but only "1.75-ble" taxation.

Footnote 2: Actually, it's not quite that simple. We assumed the new buyer's demand for 5.48% after-tax was fixed. Really, it's in comparison to other investments of equal risk. And those other investments also have tax burdens associated with them. The reason the new buyer demands 5.48% is because that's what he could get, after tax, from a bond of similar risk.

But if there were no corporate taxes, the rate of return on bonds would also be higher. Therefore, the new buyer might be demanding, and receving, a higher rate. That shows that the overall investment tax rate affects the buyer, too.

So it's not really as clear-cut as saying the original owner, the one who gets the capital gain, pays all the tax. It's a complicated thing to figure out what proportion of the corporate tax is actually "paid" by the original owner, and what proportion is "paid" by the second owner. All we know is that the proportions add up to 100%, and the total tax paid is the corporate tax plus the dividend tax.

I might be wrong here ... economists, please correct me.

--------

Capital Gains and Warren Buffett: Part I

2011-10-11T13:52:00.004-04:00

Recently, Warren Buffett noted that the rate of tax he pays on investment income (around 17%) is much less than the rate his employees pay on their earned income (around 36%). In a previous post, I argued that the comparison is meaningless, at least in the case of dividends.

Some commenters, here and at Tango's blog, criticized me for not dealing with capital gains, which they say is where most of Buffett's income arises.

Capital gains come from many different sources, which bring up different issues of fairness. So, I'll have several things to say instead of just one. My conclusion will be that there's an argument to be made for taxing capital gains at a very, very low rate, perhaps even zero.

I will argue that position is true even if you believe that tax rates on the rich are too low. I believe that even if you think the rich should be taxed, at, say, 60%, or 70%, or 80%, you should STILL favor a system where their "regular" income is taxed at a higher rate and capital gains are taxed at a lower rate.

Here goes.

1. Double taxation inside a corporation

If you buy a stock, and then sell it at a higher price, your profit is a capital gain. But, in many cases, the corporation has already been taxed on the profit that forms some or all of the gain. To tax it again at the full rate is unfair double taxation.

Suppose you buy a share of a company at $100. This year, they earn $10 in profits. They pay $3 corporate tax on the profit, and keep the other $7.

So, a year later, and all things being equal, the company is worth $107 a share. You sell your share for a $7 capital gain.

But, the piece of the company you owned actually earned $10 in profits, not $7. At a 30% tax rate, you already paid $3 in corporate tax on the profit. To tax you another 30% on the remaining $7 would be unfair. That would mean you'd only keep $4.90, and your effective tax rate would be 51%.

The concept of "horizontal equity" says that people who have the same income should pay the same amount of tax, regardless of where the income came from. If the top tax rate on employment income is, say, 40%, then the effective tax on income earned through a corporation, when you combine all the taxes, should also be 40%.

As I described in more detail in the previous post, a personal capital gains tax rate of around 15% gives the result we're looking for: you keep 85% of 70% of corporate earnings, which works out to a tax rate of 40.5%.

This is exactly the same argument as in the other post, just for capital gains instead of dividends. I realize that if you didn't like that argument, you probably won't like this one either.

----

Want a real-life example? Suppose you owned one share of Chevron.

Over the 16 years from 1995 to 2010 (.pdf), Chevron made a total profit for you of about $123. It had around a 40% corporate tax rate (I'm not sure why so high -- other companies seem to be around 30%). That means it paid around $50 in taxes, leaving $73 in after-tax earnings. It paid around $27 in dividends over that stretch, leaving $45 inside the company.

In that time, the stock went from around $25 to around $100, a $75 capital gain. More than half of that capital gain -- $45 -- is from the profit on which the corporation has already paid tax for you.

To fully tax you again on that $45 profit is not particularly fair.

----

What about the rest of the capital gain, the remaining $30 out of the $75? That probably shouldn't be taxed much either, since probably a lot of that is just inflation. Which brings us to number two.

2. Inflation

The idea behind income tax is that when you become wealthier, you give some of it to the government to provide public services. But when the nominal value of your assets goes up only because of inflation, you're not wealthier, are you?

In 1980, you bought a house for $100,000. Today, you sell it for $300,000. Are you really $200,000 wealthier? Of course not. That's just inflation increasing the price of your house. In non-monetary terms, you might have paid 200,000 loaves of bread for it in 1980 (at 50 cents a loaf). In 2011, you sell it again for 200,000 loaves of bread (at $1.50 a loaf). Really, you've broken even.

This is pretty obvious, and I think almost everyone understands this already. That's why, in both Canada and the US, they offer tax relief for capital gains on houses you live in. In Canada, you pay absolutely zero capital gains tax when you sell your primary residence. In the USA, I once read, your first $400,000 in gains is tax-free if you buy another house with it. (Is that still true?)

If the government didn't do that, there would be riots in the streets. You wouldn't be able to move! If you're living in a $500,000 house with $200,000 worth of taxes due when you sell it, you'd have to downsize substantially. That would obviously be unfair. In most cases, the profit you made on the house is artificial, just an artifact of inflation.

The same is true for, say, stocks and mutual funds. Suppose you bought a share twenty years ago for $10. It never paid dividends. Today, you sell it at $20, but because of inflation, the $20 buys only what $10 bought then.

You really haven't made a profit. Yes, you got more dollar bills now then you paid in the past, but in terms of actual wealth -- the number of loaves of bread it would buy -- you just barely got your investment back.

That's part of the reason why the capital gains tax rate is lower than the regular tax rate: to compensate for the fact that a significant portion of a capital gain isn't really an increase in wealth.

Perhaps the best policy would be that when you sell an asset, you adjust for inflation when figuring your gain, and then you pay tax on that adjusted gain. The problem with that is that it's complicated and involves lots of arithmetic. I'd support implementing it anyway.

----

Now, you might be saying, that's fine if your capital gain just keeps pace with inflation. But Warren Buffett is famous for his investing prowess, where he makes capital gains that far outstrip inflation!

To which my response is: OK, but first, can we agree that he shouldn't have to pay tax on the inflation portion of his gain? (And if we agree on that, then at least we agree on at least one reason that Buffett's capital gains rate should be less than his employment income rate, right?)

But, yes, that still leaves the non-inflation portion of Buffett's gain, which is probably still substantial. That'll be in Part II.

Would MLB salaries drop if all players were free agents? Part II

2011-10-07T13:30:00.002-04:00

In my previous post, I said I had another argument for why free agent salaries would drop if all players were free agents. Here it is.

Right now, some players are free agents. Their value seems to be about $4.5 million per win.

Now, suppose that, instead of *more* players being free agents, *fewer* players become free agents. In the extreme case, suppose that only one player is a free agent, with a value of 1 WAR. What happens?

It seems like his price will be bid up. But why? It can't be not scarcity in and of itself. Because, even in the real world, when there are lots of free agents, there is still be a time when there's only one left, and HIS price still seems to be $4.5 million per win. There's something else going on. Here's what I think it is.

In in a world with few free agents, wins must be distributed without regard to where they can make the most money. They go to whichever teams made the best draft choices or trades. That's inefficient, financially, for the league as a whole.

The Yankees value wins highly, and would like to buy more, but they can't. The Pirates don't value them much, and would like to sell some to the Yankees. But MLB rules forbid that. So the Yankees are stuck with many fewer wins than they want, and the Pirates are stuck with many more.

So what happens when this one and only free agent goes on the market? Clearly, the Yankees and Red Sox are desperate for wins, with which they can make a lot of money. They'll easily outbid the Royals and Pirates.

But why will the price wind up higher than $4.5 million? Because $4.5 million is the price that results when teams have the ability to fill a lot more of their needs. The Yankees may have been blessed with only 80 wins from their farm system, but they've been able to sign free agents to bring themselves up to 95. And $4.5 million is the value of that 95th win. The wins before that, they valued much higher (or they wouldn't have bought them).

But, in this case, the Yankees are truly stuck at 80. That 81st win they're thinking of buying must be worth more to them than the 95th win (which is worth $4.5 million). And so, they'll be willing pay a lot more for it.

The same logic applies to the Red Sox, and the Phillies, and other teams, and so the price of the single free agent gets bid up well beyond $4.5 million.

--------

That argument shows that fewer free agents means higher costs. That means that more free agents means lower costs, which is what we were trying to prove.

--------

Another thing we can conclude is that the more free agents there are, the less competitive balance. Why? Because the fewer the number of free agents, the more wins are distributed haphazardly among teams. Since teams aren't allowed to sell those wins, small-market teams wind up with wins they otherwise wouldn't have bought. That means more competitive balance.

If all players were free agents, it's possible that some teams would not find it profitable to buy ANY wins, and would stay with replacement-value, minimum-salary players. Obviously, that means competitive balance suffers.

--------

At the risk of my usual overkill, here's another way to look at it:

Suppose that there is a fixed supply of BMWs. In a free market, only rich people would own them, because they're of little use to poor people (who can't drive them much because they can't afford much gas or maintenance). There might be 100,000 rich people in the city who might fancy a nice BMW, but only 10,000 actual cars. So the BMWs go to the people who bid the most for them, and maybe the auction price is $50,000 each.

Now, suppose MLB calls half of the cars "draft choices", airdrops them randomly on households, and prohibits selling them for what they're worth. The poor people are happy to have them, since they're free, but they don't get much benefit from them. On the other hand, the rich families who got them are thrilled: some of them were about to go out and spend $50,000 on one, and now here's one for nothing!

But now, that leaves only 5,000 cars left for auction. And there might still be 99,000 rich people who are interested. Obviously, with fewer cars available on the market, but not many fewer buyers, the price goes up, maybe to $100,000.

Also, "competitive balance" increases. It used to be that the rich owned 100% of the BMWs. Now the rich only own maybe 60% of the BMWs.

And, none of this would happen if the poor people were allowed to sell their cars to the rich people. In that case, the cars would still go for $50,000, same as before, and "competitive balance" wouldn't change. If Bud Selig changed the rules so that the poor could sell to the rich, both the poor and rich would benefit: the poor people would have more money, and the rich people would get their cars cheaper.

Who wouldn't benefit? The BMW fans rooting for poor people. These fans don't care how much money their poor friend has: all they live for is the day that their poor friend drives a BMW to the World Series! Before, when their poor friend had to keep his car, they had some hope. Now that their poor friend will almost always sell his airdropped car, they have little to no hope.

-------

Another thing we can conclude is that it must be true, right now, that there do exist poor teams who have BMWs they'd like to sell but can't. In previous posts, I assumed this didn't happen, to keep things nicely theoretical. I assumed that even the Royals can earn a little bit more by buying a win or two, at the going rate of $4.5 million.

But now we have evidence that's not true, that there are some teams who want to sell wins but can't.

Why do I say we have evidence? Because the previous post showed that, today, if all players were free agents, the price would come down. This proves that there must be at least some BMWs being held by poor households. Otherwise, it wouldn't matter if you airdropped all the BMWs, or none of them: either way, they'd go to the same rich people at the same price. The fact that free-agent restrictions are increasing prices proves that MLB could make more money redistributing wins to the rich teams.

But, that makes an additional assumption: that the fans only care about wins, and not about the fairness of a sport that organizes itself so the Yankees will always be great, and the Royals will always be bad. As I once wrote, I think that even though wins seem to drive revenues today, the fans might get sick of it in the future, and MLB might be better off sacrificing some short-term revenues in favor of keeping the fans interested in the long-term.

-------

In summary, I think these arguments lead to a few real conclusions about the current state of MLB:

1. More free agents would mean lower salaries for those free agents;
2. MLB could make more money by allowing small-market teams to sell players to big-market teams;
3. A marginal win is worth an equal $4.5 million not to all teams, but only to the middle-class and rich teams;
4. The "arbs" and "slaves" do, in fact, contribute to competitive balance.

Would MLB salaries drop if all players were free agents?

2011-10-06T18:36:00.004-04:00

If every player became a free agent, would player salaries drop? Here's an argument that suggests that, yes, they *must* drop. I'm not sure if it's right, but I can't find a flaw ... maybe you guys can.

Assumptions: all teams are rational, and all teams know the expected value of the future performance of every player.

Here goes.

-----

Suppose that right now, at equilibrium, a marginal win in the major leagues is worth about $X million.

That means that the last win every team buys (in free agency) must give them $X million in extra revenues. If it gave less, they wouldn't have signed the guy. If it resulted in more than a $X million increase, the team could probably increase profits by signing even more free agents.

OK, now the hypothetical: what if EVERY player were a free agent? That is, what if there were no "arbs" or "slaves", so that every player cost the same price per WAR?

I don't know the exact breakdown of free agents vs. arbs vs. slaves. But, for the sake of argument, let's suppose that, right now, free agents contribute 1/2 the overall WAR, arbs contribute 1/4, and slaves contribute the other 1/4.

Also, let's assume that arbs make half of what free agents do, and that slaves make $0. Then, the average WAR costs major league teams exactly $0.625X -- that is, 5/8 of what a free agent WAR costs.

The fraction 5/8 works out to 62.5%. That's not too far from the fraction of revenues that go to player salaries. So, if every player were a free agent at $4.5 million per WAR, the 62.5% would go to 100%, and it would turn out that ALL of MLB's revenues would go to payroll.

That can't happen, can it? If payroll is higher than revenue for the league, it must be the case that payroll is higher than revenue for at least one team. For that team, it would be better to cut payroll. The only way it wouldn't be better to cut payroll would be if that team would lose even MORE money with EVERY possible payroll cut.

But ... minimum payroll is only about $13 million. It's hard to imagine that even a small-market team wouldn't get $13 million in revenue, even with a replacement level team expected to go 60-102. (And, in any case, the team could fold, and avoid losing money that way.)

So, no team would ever spend more on payroll than it got in revenues. Therefore, payroll can't be more than 100% of revenues. Therefore, at least one of the assumptions in our model must be false. The most likely candidate is the assumption that wins would still cost $4.5 million.

So it must be the case that the cost per free-agent WAR would drop, from $4.5 million to something lower, to allow the league to still turn a profit.

--------

Does that argument work? I think it does. But I have another one if that one doesn't.

Why Warren Buffett is wrong about tax rates

2011-10-03T12:38:00.005-04:00

Note: non-sports post. Lots of numbers, though!

-----

As you've probably heard by now, Warren Buffett thinks the rich should pay more taxes. Much of his argument is based on the fact that wealthy people pay a lower percentage of their income to the government. Buffett writes,

"Last year my federal tax bill — the income tax I paid, as well as payroll taxes paid by me and on my behalf — was $6,938,744. That sounds like a lot of money. But what I paid was only 17.4 percent of my taxable income — and that’s actually a lower percentage than was paid by any of the other 20 people in our office. Their tax burdens ranged from 33 percent to 41 percent and averaged 36 percent."

But, like a lot of numbers that get thrown around, this "percent of taxable income" is misleading. Yes, the number 17.4 is lower than the number 36. But if you look more closely, Buffett is actually paying at a comparable rate to that of his employees.

The issue that confuses things is that Buffett earns most of his income through corporate dividends (some of it also comes from capital gains, but I'll ignore those for now). Dividends are paid out of after-tax profits of corporations. That means, effectively, that the corporation has already paid most of Buffett's tax.

Suppose Buffett's employee earns $50,000, and pays $18,000 in taxes, which is 36 percent. As for Buffett himself, suppose he owns shares of McDonald's. Let's say he owns 7,609 of those shares, which correspond to the same $50,000 in McDonald's pre-tax profits. (My figures are approximate, all rounded from this Value Line summary.)

So, what happens? Well, McDonald's gets taxed at about a 30% rate. So that leaves only $35,000 in after-tax profits. Then, the company pays Buffett a dividend. It doesn't pay the entire $35,000, because it keeps some to reinvest. It pays Buffett only about half of that, maybe $17,000. Then, Buffett pays an additional 15% tax on that $17,000.

So: on his $50,000 in profits, Buffett pays $15,000 through the corporation, and then an extra $2,550 in dividend tax. That's a total of $17,550 out of $50,000, which is ... about 35%, approximately the same as his employee.

--------

Actually, that's not quite right. It actually understates Buffett's tax rate.

As we saw, McDonald's makes $50,000, pays $15,000 in taxes, sends $17,000 to Warren Buffett, and reinvests the remaining $18,000. But our calculation assumed that the $18,000 part is still Buffett's, but is fully tax paid. It's not. Eventually, Buffett will claim that $18,000 personally, either through another dividend, or through a capital gain when he sells his stock. At that point, he'll pay another 15%.

So, really, the bottom line is: Of every dollar Buffett earns through McDonald's, he gets to keep 85% of 70% of it. That's 59.5%. So his effective tax rate is 40.5%.

You can argue that 40.5% is too high, or you can argue that 40.5% is too low. But you CANNOT argue that Buffett's tax rate is only 17%. That simply isn't true in any real sense. It's true only in a misleading technical sense, in that McDonald's pays some of Buffett's tax for him. The fact that the corporation makes the payment doesn't mean that it doesn't actually come out of Buffett's pocket.

---------

This is actually a standard explanation of why taxes on dividends are lower than taxes on "work" income. In fact, it's actually the stated rationale. Canada has a complicated method of calculating taxes on dividends, a method that actually takes into account how much corporate tax was already paid. The explicit idea is that the overall tax rate should be the same, no matter if you earned the income through a direct investment, through an investment in a corporation, or through employment. It doesn't always work out perfectly, according to my accountant (who tells me it's a little higher through a corporation), but it's close.

It's well-enough known to economists and accountants and people who work in corporate finance that, like other bloggers who have written about this, I'm surprised Warren Buffett didn't know it.

Now, it could be that he knows it, but doesn't believe that the corporate taxes should "count". A lot of people somehow believe that corporations should count as separate "people," and so it's fair for both McDonald's and Buffett to pay taxes separately, where the total adds up to more than if Buffett made the money directly.

But that really doesn't make sense. The fact is that if Buffett owns McDonald's, and McDonald's earns $50,000 from his investment, that $50,000 *belongs to Buffett*, even if, right now, it's classified as corporate earnings. Corporations, and their earnings, are the property of their owner, and Buffett is that owner. The fact that the first $15,000 of taxes appears on the corporate tax return instead of Buffett's absoutely does not change the fact that Buffett is paying that tax.

If you still don't agree with that, if you think that Buffett really isn't paying that 30%, then you might like these two options I'm about to show you. Those could reduce personal taxes substantially, while still providing the same amount of money to the government!

Here's number 1. As of tomorrow, ever person in the country has to start up a corporation, of which he/she is the sole shareholder and CEO. Also, all employers now have to pay any salary to the corporation.

So, what happens is this: your corporation now pays 30% of its income -- your salary -- in corporate tax. It then pays you the rest as a dividend. Like Warren Buffett, you now pay only 15% in taxes. Indeed, you might pay zero in taxes -- according to this Wikipedia page, low-income Americans pay 0% tax on dividends!

That means that Warren Buffett's employees will now definitely have a lower tax rate than he does -- or at worst, the same rate -- because they are taxed exactly the same way!

Would you support that new law? You probably wouldn't. You'd see that that was just a sneaky way of taxing people the same rate as always, but making it look like they're not paying much tax. That's exactly what I'm arguing in the case of McDonald's.

--------

Here's number 2. Right now, people take home a lot less than their salary, because of payroll deductions for income tax. Someone making $50,000 might actually take home only $35,000.

So here's what we do. We eliminate payroll deductions and personal income tax completely. Instead, we implement a corporate payroll tax, which works out to exactly the same as the income tax used to. So, now, your employer still gives you $35,000 to take home, but pays an extra corporate tax of $15,000. You pay zero percent tax on the $35,000.

Perfect, right? Now the income tax on "work" is zero percent. Warren Buffett is still paying 17.4% on his investments. Situation resolved!

I bet you think that's ridiculous. You probably should. Whether you pay the tax, or your employer pays the tax, it's the same thing: you do $50,000 of work, and the government gets 30% of it.

Well, it's the same thing for McDonald's profits. Whether McDonald's pays the tax, or Warren Buffett does, or they both do, the fact remains: McDonald's makes $50,000 of profit, and the government gets 40.5% of it. How much of that 40.5% comes from a cheque from Buffett, and how much comes from a cheque from McDonald's, doesn't matter.

--------

OK, here's one more suggestion that's less ridiculous.

Change the law a bit, so that when McDonald's pays a dividend to an American taxpayer, they don't pay tax on that part of their profit. That sounds kind of fair, right? If McDonald's doesn't get to keep it, they don't pay tax on it. Just like they can deduct interest that they pay to a bondholder, they can deduct interest that they pay to a shareholder.

Then, when the shareholder receives the dividends, tax them at the normal rate, as if they were "work" income.

Sounds reasonable, right? Well, it works out to almost same amount of tax collected. Actually, if Buffett's personal tax rate is around 33%, you can leave out the "almost" -- it's exactly identical.

McDonald's makes $50K. They send Warren Buffett $21,428.57. Buffett pays about 33% of that in taxes, or $6,978, leaving him $14,450. After paying Buffett, McDonald's has $28,571.43 left. They pay the government 30% of that, or $8,571.43. That leaves them $20,000 to reinvest.

That's EXACTLY what's already happening, in today's system where Buffett *appears* to be only paying 15%:

-- McDonald's makes $50,000
-- Buffett keeps $14,450 after taxes
-- McDonald's keeps $20,000 after taxes
-- The government gets $15,550.

One way you look at it, it looks like Warren Buffett pays only 15% in taxes. Another way, it looks like he pays 40.5% in taxes. A third way, it looks like he pays 33% in taxes. But the results, all three ways, are exactly the same!

No matter how you figure it, the bottom line is the same. Buffett gets 28.9% of the profit, McDonald's keeps 40% of the profit, and the government gets 31.1% of the profit. The difference is how you do the accounting. If you're a government that wants to make it look like the rich have it too good, you levy the entire 31.1% on the corporation, so that it looks like Buffett pays zero. If you're a government that wants to make it look like corporations aren't taxed enough, you levy the entire 31.1% on Buffett, so that it looks like McDonald's pays zero.

In a vacuum, Warren Buffett's supposed 17% tax rate doesn't mean anything. It's an artifact of how you do the accounting. To really see what's happening, you have to look at the end result. And that end result, in this example, is that the government winds up with 31.1% of the money that would otherwise have been Buffett's.

You may think that's too low. You might think that's too high. That's fine. But either way, the "17%" figure is not relevant, even if Warren Buffett thinks it is.

How good are sports pundits' predictions?

2011-09-27T11:17:00.006-04:00

According to this Freakonomics post, the experts who make NFL predictions aren't very good at it. Freakonomist Hayes Davenport checked the past three years' worth of predictions from USA Today, Sports Illustrated, and ESPN. He found that the prognosticators correctly picked only 36% of the NFL division winners.

That doesn't seem that great. Picking randomly would get you 25%. And, as the post points out,

"if the pickers were allowed to rule out one team from every division and then choose at random, they’d pick winners 33% of the time. So if you consider that most NFL divisions include at least one team with no hope of finishing first (this year’s Bengals, Chiefs, Dolphins, Panthers, Broncos, Vikings, and Manning-less Colts, for example), the pickers only need a minimum of NFL knowledge before essentially guessing in the dark."

Well, it sounds right, but you have to look deeper.

Winning depends on two things: talent, and luck. Since luck is, by definition, random, when you predict a winner, the only thing you can do is pick the team with the most talent. And, despite the 36% figure, there's no evidence that the pundits misjudged the talent. Because, just by luck, sometimes the best team won't win, and that's unpredictable.

Two days ago, the Buffalo Bills upset the New England Patriots, despite being 7:2 underdogs (I'm estimating 7:2 based on the 9-point spread). What percentage of experts would you expect to have got that right? Your answer should be zero percent. Nobody with any knowledge of football should have thought the Bills had a better than 50 percent chance of winning. On the off-chance that you DO find someone who picked the Bills to win, he's probably a crappy predictor -- maybe he just flips a coin all the time.

In the case where the underdogs wind up winning a game, or finishing first in their division, the truth is the opposite of what Freakonomics implies. In that case, the higher percentage of correct predictions, the WORSE the pundits.

So what does that 36% figure actually tell you? By itself, absolutely nothing. You have no idea, looking at that bare number, how good the pundits are. It depends. If it was *all* bad teams that won, a number as high as 36% means the experts are wrong a lot, but 36% of their bad picks happened to turn out OK. If it was all good teams that won, a number as low as 36% means the experts are wrong a lot -- they must have picked 64% bad teams. And, if it was exactly 36% of the best teams that won, but those aren't the cases where the experts were right, then, again, the experts are wrong a lot.

But, if it was exactly 36% of the best teams that won, and those are exactly the cases where the experts were right ... then the experts are perfect predictors.

So you can't just look at a number. 36% may be bad, but it might be awesome. It depends what actually happened.

-------

However: while this logic applies to picking outright winners of games or divisions, it doesn't apply to picking against the spread. Why not? Because, against the spread, the presumption is that the odds are close to 50/50.

In the Patriots/Bills game, the odds were roughly 77/22. Some experts might have pegged the Bills as having a 25% chance of winning, while some may have estimated only 20%. Still, both pundits would have obviously still bet on the Patriots. The fact that New England ended up losing isn't really relevant.

But against the +9 spread, you might have a reasonable difference of opinion. One expert might figure the true spread should be +8.5, and another might figure +9.5. So the first guy takes the Bills, and the second takes the Patriots.

On bets that are approximately 50/50, reasonable experts can disagree. On bets that are 77/22, they cannot. So, when it's 50/50, a higher percentage could, in fact, mean better predictions.

Still, there's lots of luck there too. If a pundit predicts all 256 games in a season, the (binomial) standard deviation of his success rate will be a little over 3 percentage points. So one predictor in 20 will be over 56%, or under 44%, just by luck.

That means it's still hard to figure out who's a "better" expert and who's not.

--------

The post goes on to criticize the pickers for risk aversion. Why? Because, it seems, the experts tended to pick the same teams that won last year.

Um ... why is that risk aversion? It stands to reason that the teams that won before are more likely to still be pretty good, so they're probably reasonable picks. But, the author says,

"Over the last fifteen seasons, the NFL has averaged exactly six new teams in the playoffs every year, meaning that half of the playoff picture is completely different from the year before. ... Given that information, a savvy picker relying on statistical precedent would choose six new teams when predicting the playoffs."

That doesn't follow at all! Just because I know six favorites will lose doesn't mean I should pick six underdogs! That would be very, very silly.

It's like predicting whether John Doe will win the lottery this week. The odds say no, and that's the way I should bet. And it's the same for Jane Smith, or Bob Jones. If there are a million people in the lottery, I should pick them all to lose. I'll be right 999,999 times, and wrong once.

But, according to Freakonomics, I should arbitrarily pick one person to win! But that's silly ... if I do that, I'll almost certainly be right only 999,998 times!

It's not exactly the same, but this logic reminds me of the Jeff Bagwell prediction controversy. In the fall of 1990, Bill James produced forecasts of players' batting lines for 1991, and it turned out that Bagwell wound up with the highest prediction for batting average. It was said that Bill James predicted Jeff Bagwell to win the batting title.

But, obviously, he did not. At best, and with certain assumptions, you might be able to say that James had Bagwell with the *best chance* of winning the batting title. But that's different from predicting outright that he'd win it.

Back to the lottery example ... if I notice that John Doe bought two lottery tickets, but the other 999,999 people only bought one, I would be correct in saying that Doe has the best chance to win. That doesn't mean I'm *predicting* him to win. He still only has a 1 in 500,000 chance.

--------

Finally, in their introduction to their post, Levitt and Dubner say,

" ... humans love to predict the future, but are generally terrible at it."

I disagree, especially in sports. Yes, very few people can beat the point spread year after year. But that doesn't show that the experts don't know what they're doing. It shows that they DO! Because, after all, it's humans that set the Vegas line, the one that's so hard to beat!

I'd argue 180 degrees the opposite. In sports, humans, working together, have become SO GOOD at predicting the future, that nobody can add enough ingenuity to regularly beat the consensus prediction!

The Bayesian Cy Young

2011-09-22T12:54:00.004-04:00

At Fangraphs, Dave Cameron and Eric Seidman have a nice discussion (hat tip: Tango) on who's the better Cy Young candidate: Clayton Kershaw, or Roy Halladay?

Part of the discussion hinges on BABIP: batting average on balls in play. As Voros McCracken discovered years ago, pitchers generally don't differ much in what happens when a non-home-run ball is hit off them. Most of the overall differences between pitchers, then, are due to the fielders behind them, but mostly due to luck.

So far in 2011, Clayton Kershaw has a BABIP of .272, which Eric decribes as "absurdly low." Still, Eric thinks it might actually be skill rather than luck, since since .272 it's not that much different than Kershaw allowed in previous years. Dave argues that Kershaw's three seasons is still a fairly small sample size, and points out that most of his BABIP advantage comes from his record at home (he's about average on the road).

Anyway, my point isn't to weigh in to which one is right -- they do a fine job hashing things out in their discussion. What I want to talk about is something they both seem to agree on: that it's important whether the BABIP is luck or skill. If it's luck, that reduces Kershaw's Cy Young credentials. If it's skill, he's a better candidate.

Seems reasonable, and I don't necessarily disagree. But let's see where that logic leads.

Because, there are other kinds of luck, or factors that pitchers can't control. For instance, there's park (which is usually already adjusted for in WAR, the statistic Eric and Dave cite most in this debate).

There's also quality of opposition batting. It's probably not too hard, if you have good data, to figure out how much either of the pitchers gained by being able to pitch to inferior hitters. You could also check if one of them had the platoon advantage more often. And, if one of them pitched more at home than the other one did.

We'd probably all agree, right, that you'd want to adjust for those kinds of things if we had the information? To be clear, I'm not criticizing Dave or Eric for not spending hours figuring this stuff out. I'm just saying that if you have the data, it's relevant in comparing the pitchers.

There are other things too, that eventually we'll be able to figure out, that we can't right now because (as far as I know) the research hasn't been done. Suppose Kershaw throws a pitch at a certain speed, with a certain break, on a certain count. And, someday, we'll know that kind of pitch is swung on and missed 30% of the time, called a ball 5% of the time, called a strike 10% of the time, fouled off 10% of the time, and hit in play 45% of the time with an OPS of .850. Maybe, overall, that pitch is worth (say) +0.05 runs (in favor of the pitcher).

Once we have that kind of information, we can check for "batter swing luck". If it turns out that batters just randomly happened to go +0.03 on that pitch from Kershaw this season, instead of +0.05, we should credit him the extra 0.02, right? He delivered a certain performance, and the batters just happened to get a bit lucky on it, as if his BABIP was too high. (This measure would probably substitute for BABIP: it includes balls in play, but also home runs, swings-and-misses, and walk potential.)

So we'd adjust Kershaw and Halladay for how lucky the batters were on those swings.

That's not unrealistic, and it'll probably eventually happen, to some degree of accuracy. Here's one that probably won't, at least not for a few decades, but it works as a thought experiment.

Imagine we hook a probe to every batter's brain, so on every pitch we can tell if he's guessing fastball or curve, and if he's guessing inside or outside. After a couple of years of analyzing this data, we figure that when he guesses right, it's worth +0.1 runs (for the batter), when he guesses half-right, it's worth 0, and when he guesses wrong, it's -0.1.

That again, is something out of the control of the pitcher (especially if both batter and pitcher are randomizing using game theory). So you'd want to control for it, right? If Halladay is having a good year just because batters were unlucky enough to guess right only 23% of the time instead of 25%, you have to adjust, just like you'd adjust for a lucky BABIP.

This will change the definition of "batter swing luck," but not replace it. First, the batter may have been lucky enough to guess right, which is worth something. Then, he might have been lucky enough to get better than expected wood on the ball even controlling for the fact that he guessed right.

So you've got lots of sources of luck:

-- park
-- day/night
-- distribution of batters
-- platoon luck
-- BABIP luck
-- batter swing luck
-- batter guess luck

You'd want to adjust for all of these. Right now, as I understand WAR, we're adjusting for park and BABIP.

What about the others? Well, we can't really adjust for those. We *want* to, but we can't.

So, we make do with just park and BABIP. Still, no matter how many decimal places we go to with the debate on Kershaw/Halladay, we're still only going to have our best guess.

At least we can argue that if all the other things are random, we should still be unbiased. Right?

Well, not really. From a Bayesian standpoint, we have a pretty good idea who had more luck. It's much more likely to be Kershaw.

Why? Because Halladay's performance is much more consistent with his career than Kershaw's. Kershaw's a good pitcher, but wasn't expected to be *that* good. Halladay, on the other hand, is having a typical Halladay season. Well, a bit better than typical, but not much.

I'd be willing to bet a lot of money that if you found 50 pitchers who had a better-than-career season, by at least (say) 1.5 WAR, you would find that those 50 pitchers had above-average BABIP luck. It stands to reason. I won't make a full statistical argument, but here's a quick oversimplification of one.

A pitcher can have his talent go up or down from year to year. He can have his luck go up or down from year to year. That's four combinations. Only three of them are possibly consistent with a big improvement in WAR: talent up/luck up; talent up/luck down; talent down/luck up. Two of those have his luck going up. So, two times out of three, the pitcher was lucky.

The argument applies to *all* sources of luck. Even after taking BABIP into account, if a pitcher's adjusted performance is still above his career average, he's still more likely to have had good luck than bad, in other ways (batter swings, say).

I don't have an easy way to quantify this, but still I'd give you better-than-even odds that, stripping out all the above, Halladay is performing better than Kershaw -- even after adjusting for park and BABIP.

If you have two players with similar, outstanding performances, the player with the better expectation of talent is probably the one who's actually having the better year. To believe that Kershaw was really likely to have had a better year than Halladay, you really need him to have put up *much* better numbers. Either that, or you need a way to actually work out all the luck, and prove that the residual still favors Kershaw.

I should emphasize that I am NOT talking about talent here. I think most people would agree that Halladay is still more talented than Kershaw, but would nonetheless argue Kershaw might still be having the better season.

But, what I'm saying is, no, I bet Kershaw is NOT having a better season, even if his numbers look better. I'm saying that it's likely that Kershaw *is actually not pitching better*. If we had the data, it's more likely than not that we'd see that batters are just having bad luck -- not only are they (perhaps) hitting the ball directly to fielders, as BABIP suggests, but they're probably swinging and missing at hittable pitches.

---------

Another way to look at it: if two pitchers have mostly the same results, but one has better stuff, what does that mean? It means that the pitcher with the better stuff must have been unluckier than the pitcher with the worse stuff. In other words, the batters facing the better stuff must have been luckier.

We don't know for sure, of course, that Halladay had better stuff than Kershaw. But history suggests that's more likely. And so, the odds are on the side of Kershaw having been luckier than Halladay. How much so?

I don't know. One mitigating factor is that Kershaw is young, so you'd expect more of his improvement to be real. But, still, a small improvement is more likely than a large improvement, so the odds are still on the side of postive luck over negative luck.

---------

Does that take some of the fun out of the Cy Young? I think it certainly does make it a little bit less entertaining, at least until we have better data. That's because, as long as we remain ignorant of a significant amount of luck, it requires a much bigger hurdle to award the honor to anyone other than Halladay.

This is a bit counterintuitive, but it's true. Suppose a good but not great pitcher -- Matt Cain, say -- has almost exactly the same stat line as Roy Halladay, including BABIP, but is actually better in some categories. Perhaps he a couple of extra strikeouts, and a couple fewer walks.

From the usual arguments, there would be absolutely no debate that Cain's season is better, right? He's better than Halladay in some categories, and the same as Halladay in all the others.

But ... if you're trying to bet on which player actually pitched better after removing all the luck, you'd still have to go with Halladay.

-----

UPDATE: on his blog, Tango writes,

Aside to Phil: Marcel had Kershaw with a 3.07 ERA for 2011, and Halladay at 3.04. So, while you make great points in your article, you didn’t have the right examples! Sabathia and Verlander would have been better examples.

Oops! I'll just leave it the way it is for now, but point taken.

Stock market integrity and the OSC's bizarre Catch-22

2011-09-18T14:29:00.011-04:00

Warning: non-sports, non-numbers post. Has to do with securities regulation and put options and bureaucratic illogic. Still, should be comprehensible to all.

-------

The Ontario Securities Commission (OSC) is a government body that regulates capital markets (i.e., stocks, bonds, options, etc.). It declares, among others, a responsibility to "foster fair and efficient capital markets and confidence in the markets." Recently, it made a decision that seems so obviously unfair and wrong that it has the opposite effect -- I am now materially *less* confident of the integrity of the market than I was before. It's not just the precedent this decision sets, but my fear that the OSC just doesn't get it. What unfair decisions will they make next, and is my retirement portfolio in jeopardy?

Of course, I might be wrong in my logic. Please correct me if I am. If you're more up-to-date than I am in how securities regulations work, let me know and I'll post corrections.

I'm going to start with an analogy that illustrates the issue.

--------

I buy a house and a piece of land, for $400,000. I insure the house. There is a regulation on the books, quite reasonable, that it is not legal to sell land that is known to be contaminated, or to sell a house that is known to be uninhabitable. But the house and land are fine, and the sale goes through.

Later, an arsonist burns down half the house and contaminates the land. The state comes in and begins an investigation.

I contact the insurance company. They agree I'm covered for $400,000. They prepare to cut me a cheque.

But, before anything else can happen, the regulator steps in. "You can't do that," they say. "When you settle an insurance claim, it means the house and the land transfer to the insurer. But the house is uninhabitable, and the land is contaminated. So, you can't transfer the house. Therefore, the settlement is illegal."

In any case, the insurer doesn't have to pay. He walks away happy, even keeping my premium, and I'm stuck with the loss.

It's a kind of Catch-22.

---------

Not fair, right? And *obviously* unfair.

What's happened is that the regulator is blindly sticking to a regulation that's not always right. Sure, it might be a good idea to prohibit the sale of contaminated land *as a general rule*. But there are exceptions. This is an exception. In fact, it's an exception where it's exactly the opposite -- where it's absolutely WRONG to prohibit the sale.

It's like, "don't jump out of the fifth floor window, you'll die." Sure. But if the building is on fire, and the smoke is choking you, and the firemen are holding a net below and yelling at you to jump ... then the rule reverses. "Don't NOT jump out the fifth floor window, you'll die."

---------

So here's the real story, which follows the analogy quite closely. Some background first.

Sino-Forest is a Canadian forestry company that does all its business in China. Its stock went from $1 to $24 over the last ten years or so, as it grew and bought forests in China for harvesting. In June, a small company called "Muddy Waters," run by a man named Carson Block, put out a report alleging, with evidence, that Sino-Forest was a fraud -- it didn't really own the forests it claimed it did. The stock dropped immediately from $24ish, and fluctuated between $5 and $8 for the next two months.

The company claimed innocence and hired independent auditors, but no information was forthcoming and the company cited documentation delays. The stock dropped further. At one point, the OSC said it was investigating.

Finally, in August, the OSC claimed there was evidence of fraud. They did not give details, and speculation is that they got their information from the auditors, and from Muddy Waters. The OSC immediately prohibited further trading in the stock, and ordered the CEO to resign.

A few hours later, the OSC was told it didn't have the right to order any resignations. Belatedly realizing it had overstepped its authority, the OSC retracted that part of its order. Nonetheless, the CEO voluntarily stepped down a few days later.

In addition to shares of stock, there were also "put options" trading on Sino-Forest. A put option is a contract between two parties. One party pays the other some money -- say, $1 per share -- and, in return, receives the right (but not the obligation) to force the other party to buy his shares by a certain date, at a certain price. Say, $20 by August 19.

The idea is that you can use a put option as insurance. If you own 100 shares, with a value of $2,000, and you're scared the price will drop, you can buy 100 put options for $100. Then, even if the stock drops, you know you can still get $2,000 for them on August 19.

It's exactly like insurance on a house. You pay $100 for the insurance, and if anything bad happens to your house between now and August 19, the insurance company will take the house away and give you a $2000 settlement.

So, at this point you can guess what happened next. The OSC prohibited the contracts from being exercised. The OSC said, "I don't care if you bought the insurance. Settlement means that you would have to sell the shares to someone else, and we've prohibited you from doing that."

And, of course, August 19 has come and gone. (There are contracts with other expiry dates too -- over different months -- but August 19 was one of them.) The contracts have expired and are now worthless. The OSC blindly followed the rule "it's bad for markets if shares of a fraudulent company are bought." That's not always true. In this case, it's WORSE for the market if the shares are NOT bought.

Normally, people don't want to buy a company when it might be a fraud. In this special case, people DO want to buy a company ONLY when it's a fraud.

This is so obviously wrong that anyone should understand that it's unfair. But, especially, the OSC, which is the regulator, and supposed to be an expert in markets, and how they work, and investor confidence ... how did they make a mistake like that?

Not only is it unfair, but ... if this precedent holds, the entire market for put options falls apart. How do they not get that?

Unless it's me that doesn't get it. Which is certainly possible. If you're a Bayesian, or even if you have normal common sense, you're probably asking yourself: who's more likely to be grossly wrong: Phil, the amateur investor, or the expert regulators at the OSC? If you're Bayesian, you should probably figure that must be me who's wrong, especially when I tell you that I was unable to find anyone in the financial press complaining about any of this. To my knowledge, I'm the first and only one.

But ... I just can't see how this could be right.

---------

Well, this past week, they held a hearing to revisit the decision. I thought they'd say, "oops, sorry, we screwed up," and fix it. But confronted with all the arguments (as I presume they were), they STILL didn't get it. They only "fixed" part of it.

What they did was to say, if you already own the shares, then, OK, you can sell them to the other party for the $20 to complete the transaction and collect on your insurance. But if you happen to have the insurance contract, but you don't have the shares, because you were meaning to buy them later, then you're still SOL. You can't go out and buy the shares from someone else, so that you can collect on your investment. Instead, you have to let your options expire worthless.

Their logic appears to go something like this: "The put option is a contract to sell shares. So we'll make an exception and let you sell shares if you have a previous contact. But we won't let you BUY the shares to sell, because you only have a contract to sell, not to buy. Besides, if you bought a contract to sell, but didn't own any to sell, you're just speculating, so we don't feel much sympathy for you."

But, that's ridiculous. It's a common investment strategy to buy put options on stocks you don't own, if you expect them to drop. Sometimes it's straight speculation that there's fraud, but sometimes it's part of a more complex hedging strategy. Maybe you own a business in China, and you want to insure against a bad Chinese economy, and the easiest way is to buy puts on Sino-Forest. If China goes downhill, and Sino-Forest with it, you buy the worthless Sino-Forest shares, and sell them according to your contract, which gives you the insurance money you need.

(* In any case, since when is speculation something that anyone should be trying to avoid? Speculation is a good thing, as economists will assure you. And, securities regulators, being experts in how capital markets work, know that. Speculators keep the market liquid and efficient, moving prices closer to their true value. I personally would be hesitant to invest without speculators. Right now, I can be pretty sure that I'm paying a fair price for any stock I buy -- if the price were too high, speculators would have stepped in before and sold short to push the price down. Without speculators, I'd be more likely to be getting ripped off.

( But I digress. Oh, and while I'm digressing, a disclosure: I own shares of Sino-Forest, but have never had any Sino-Forest option positions.)

What the OSC has done with its fix is actually worse than what it did originally. It said, "we'll let you enforce your contract if we approve of your investment strategy, but we will screw you around if we don't." That's something the OSC has no business doing, favoring some parties but not others based on the capricious illogic of its bureaucrats. It's also the worst thing you can do for market confidence -- signalling to the world that the rules are unpredictable based on how the regulator feels about you.

For my part, I have bought put options before, on companies I thought were grossly overvalued. I'll be damned if I'm going to do that again, at least in Canada.

Bob McCown on "puck luck"

2011-09-16T10:54:00.005-04:00

I just got a copy of "McCown's Law -- The 100 Greatest Hockey Arguments." I'm only on number 1, but already Bob McCown nails it:

" ... hockey is enveloped by a culture that demands that everything be rationalized or explained ...

... it's hilarious the way fans react when their team loses a close game. You'd swear the players couldn't do anything right. And yet, when the same team wins a game by a one-goal margin, it's showered in platitudes.

So here's an experiment I'd love to perform sometime.

Let's take the tape of a five-year-old NHL game -- any game -- in which the score ended 3-1. Now, let's edit out the goals and leave all the rest, so that about 59 of the 60 minutes are there to watch.

Now show it to an audience of hockey fans and see if they can guess who won.

I bet they couldn't, because aside from the moments in which the goals are scored, an awful lot of hockey games are nothing but back-and-forth flow, the trading of chances and puck luck.

To have some fun, let's try the same experiment with a bunch of reporters. Then, let's show them the stories they wrote about that exact game.

Most nights in hockey, both teams skate hard, check hard, and go to the net ... And one of them has a puck hit the post and bounce into the net. And the other hits a post and watches it bounce wide. On more nights than you'd believe, the difference is as simple as that ...

In fact, I would say that puck luck, as it is often called, decides roughly half of the close games in the National Hockey League."

Absolutely right.

I'm looking forward to the rest of McCown's book. I'll probably find more things to post about later, if the quality of the first chapter is any indication.

-----

Well, one picky point on McCown's essay: I don't know what "decides roughly half of the close games" means. If a team wins 2-1, what does it mean that luck decided it, or not? That's a bit vague. I know what McCown means to say, and I agree with it on a gut level, but ... I'm not comfortable with phrasing it that way, because I like to have a precise definition.

So let's arbitrarily make one up.

Suppose you did something like what McCown suggested -- you edited a tape of the game to remove the results all the shots and "dangerous" scoring chances (that might or might not have resulted in shots). Then you somehow computed the win probability based only on the situations that appear on tape. Maybe you give a breakaway an expectation of 0.3 goals. And for a point blank slot possession, you assign 0.5 goals. And a slapshot from the point, 0.1 goals. And so on.

You compute an expected score based on that.

Then you look at the real score.

1. If the "wrong" team won, it must have done so by "puck luck".

2. If the "right" team won, but its expectation was to win by less than one goal, then you define that as a win by "puck luck".

That might actually be possible to partly figure out. The NHL website gives all the shots, by distance and type, and Alan Ryder has done lots of research on how to get scoring probabilities for shots. However, the NHL doesn't list other kinds of scoring chances aren't listed, so you'd have to stick to *shot* "puck luck".

In any case, even if you had scoring chances, there would still be luck unaccounted for, in the development of the play. A breakaway might have itself been caused by a defender missing an easy puck. A good chance was caused by three low-odds passes that happened to click. And so on.

So, let's try again. How about, a game is decided by "puck luck" if:

You edit the game per McCown's suggestion, and show it to reporters. You make them bet their own money on who won, against each other at odds that they negotiate. If the overall odds wind up between 60:40 and 50:50, or the overall underdog won the game, then that's a game decided by "puck luck".

I'm not suggesting you actually do this, but that you do a thought experiment and estimate what would happen. There are obviously some games where one team absolutely dominates (and wins). The reporters would obviously get the right answer here ... they'd need 90:10 odds or something to back the underdog. But there are obviously games that would look like toss-ups.

Any other suggestions for how to define that in a way where we could actually talk about how to get an answer?

--------

As an aside, I think this kind of "replay" technique has all kinds of sabermetric applications. To evaluate referee performance, take a tape of the foul, do some digital processing to obscure the players and teams involved, and get referees to judge it. To scout a pitcher, you can avoid being biased by the result of the pitch (a good pitch can still be hit for a home run) by digitally removing the result (and perhaps extrapolating/animating the last few inches, if the pitch was actually contacted). And so on.

I think I proposed this thought experiment once, along the same lines. Suppose you had a time machine. You go 40 years into the future, and you go to MLB.com, and you download video of every inning of every game.

You take 10 players across the spectrum of hitting talent: the equivalent of Albert Pujols, the equivalent of John McDonald, the equivalent of Ichiro Suzuki, and so on. You carefully select 200 AB from each of them, so that those 200 AB show the same batting line for each player, and put them on tape.

Then, you bring those tapes back to the present day, and show them to all the scouts. Would the scouts be able to tell the good players from the bad players?

Logical thinking

2011-09-13T18:07:00.004-04:00

Great example of statistical logic, from John D. Cook via Alex Tabarrok:

"During WWII, statistician Abraham Wald was asked to help the British decide where to add armor to their bombers. After analyzing the records, he recommended adding more armor to the places where there was no damage!"

Explanation at the above links.

On inequality of wealth

2011-09-08T23:11:00.005-04:00

Note: Non-sports post.

----

In the United States, the top 10% of the population earns 30% of the income. And the top 10% of the population owns 70% of the wealth.

Statistics like these seem to be popping up all over the place lately ... someone I know posted one on Facebook a few days ago, and there was a newspaper article or two in the last month. I'm not sure what happened to bring all this up. (If anyone has links from the last week or two, let me know. I can't find them at the moment.)

A couple of years ago, taking about the Gini Coefficient, I made a bunch of arguments about why the distribution of income doesn't matter much. I think it matters a bit, but not much.

Here, I'm going to concentrate on the distribution of *wealth*. For wealth, I'm going to argue that, given a particular distribution of income, the distribution of wealth is almost completely meaningless as a moral issue, or an issue of people's well being. That is: criticize, if you want, the fact that the top 10% get 30% of the income. But given that income distribution, *it doesn't matter* how much of the wealth the top 10% own: whether it's 10%, 30%, 70%, or 99%.

-----

The difference between income and wealth is that income is a rate, how much you earn in a particular year. Wealth is the total amount that you possess at a specific time.

How does anyone gain wealth? Other than inheritance (which we'll disregard here), you have to save or invest some of your income. You can earn ten million dollars one year, but if you blow it all on cocaine and hookers, your wealth will be zero.

So your wealth is a result of three things: (1) your income, (2) the amount you save, and (3) rate of return on the amount you save. As I said, if you hold (1) as fixed, wealth is affected by only (2) and (3).

Suppose you have two people, John and Mary. They have exactly the same education, and they graduate into exactly the same job, paying $50,000 a year. John spends all his money every year. Mary saves an annual $6,000 in a retirement fund, earning 5%, and spends the rest.

What happens? After 40 years, John has $0 in wealth. Mary has $725,000.

Is it fair to complain about that? I don't think so. Sure, Mary is now (fairly) wealthy while John has to live on just Social Security. But, in the past, John lived much better than Mary, to the tune of $240,000 -- $6,000 a year for 40 years. Some people's tendency would be to take some of Mary's money and give it to John. But that wouldn't be fair. It would actually be quite an injustice. Mary deliberately lived significantly worse than John for 40 years, just so she could have a better retirement. Giving that money to John would *compound* the inequality, wouldn't it? It would take from the (formerly) poor lifestyle and give to the (formerly) rich lifestyle. It would compensate for the future where Mary spends more than John, but not compensate for the past, when John spent more than Mary.

Really, even though Mary has more money than John, over their lifetimes, they're equal. Thirty-five years ago, John spent $4,000 on a new state-of-the-art TV. He knew, when he bought the TV, that $4,000 then would be the equivalent of $22,000 at retirement. He bought the TV anyway. Nothing wrong with that. He chose, freely, to live $4,000 richer than Mary back then, in exchange for living $22,000 poorer than Mary later. Mary also knew the terms of the trade, and made the other choice.

But, over their lifetime, they are exactly equal. $4,000 can buy a lot of things: a vacation, a TV, a boat, a motorcycle, or a retirement fund of $22,000. If John had bought a TV, and Mary had bought a boat, could anyone argue that Mary is richer than John because she has a boat? Of course not -- because, by the same token, John has a TV of equal value.

The same thing applies here: if John has a TV that costs $4,000, and Mary has a $22,000 retirement fund, which also costs $4,000 ... they must be equally rich, right?

-----

When you talk about the distribution of wealth, what you're really talking about, for the most part, is the distribution of a desire to save. And there is no "proper" distribution for that, any more than there's a "proper" distribution of religious beliefs. People are diverse, and they have different tendencies. Some people like to spend, and some people are compulsive savers. Humans choose differently from each other.

Suppose the 1% of the population that owns the most rare baseball cards happens to own 70% of the rare baseball cards. It just means that the other 99% don't care as much about baseball cards. If they have the same income, they just own more other things instead.

Now, you can still argue that the reason it's a problem that the top 10% has 70% of the wealth is that the bottom 90% doesn't earn enough money to be able to save. But that argument is better made by arguing about the *income* distribution. Because, otherwise, you're combining two issues: having money, and choosing to save it. If you were to complain that the top 1% own 70% of the baseball cards *because they have a higher income*, you'd be mostly wrong. Yes, the top 1% of baseball card owners probably DO have a higher income. But that's not the main reason they own 70% of the baseball cards. The *main* reason they own 70% of the baseball cards is because they really, really like baseball cards.

-------

Here's a model, for a numerical example.

Start by assuming a population of 10,000 people. They all have exactly the same education, and they all graduate at age 25 into a job that pays $40,000 a year. They work until they're 65, at which point we measure their wealth.

But they're not all the same, because they have different personalities, and characteristics, and desires. Specifically:

1. They vary in how much money they like to spend. The mean of the population is to spend 90% of their salary and save 10%, but with a standard deviation of 15 points. Nobody saves more than 50% of their salary, or spends more than 115%.

2. They vary in how many children they want, and when. 20% of them want no kids. 20% of them want one kid early in life (age 27), and 20% want one kid later in life (age 35). 20% of them want two kids early, and 20% want two kids late. Kids cost $5,000 a year in expenses to age 18, and then $20,000 annually for the next four years, all of which comes out of saving.

3. They vary in how good they are at investing their money. Some play it safe, and some are more aggressive. Some study investing, and some don't. The average annual return is 4%, with an SD of 1.5 percentage points.

4. They vary in how much effort they put into their job, which affects their annual salary increases. The mean increase is 2% a year, with a standard deviation of 0.5%. Nobody ever gets fired or earns less than $40,000.

5. Nobody goes into debt more than $50,000. Once they reach $50,000, they cut their spending to keep the debt at $50K. All debt is paid off in the year before retirement. Debt earns interest of 10%.

Under these conditions, I ran a random simulation of the 10,000 people.

So, at age 65, what percentage of total wealth will the top 10% own? Take a guess before reading on. I'll write it cryptically so you don't see it by accident when you're thinking.

Ready?

The top 10% of these graduates own (7 * 9 - 22)% of the total wealth.

Got that? It's not as big as the real-life answer of 70%, but it's pretty big nonetheless. And it's *completely* due to the decisions of the individuals themselves. There is no inequality, no racism, no bad schools, no corruption, no government favors, no explotation by greedy employers. It's just natural variation in how human beings choose to live their lives.

----------

Some of the other results:

The top 1% had 7% of the wealth.

The top 10% had 41% of the wealth.

The top 50% had 99% of the wealth.

As you would expect, the wealthiest people were the ones who saved the most and got the highest rates of return. The wealthiest, person number 7,490, wound up with just over $4,000,000 in wealth. She saved 41% of her salary and earned 8.3% per annum. In case you think 41% is a lot ... it's not, really. There are a lot of misers in the world. At retirement, this person earned $55,551, which means she was living on around $33,000 per year. That's not unreasonable for an outlier, just over 2 SD from the mean.

Overall, it turned out that number and timing of children didn't matter much. Neither did salary (although the salaries were all pretty close). Some of the richest people earned below-average salary increases.

So what mattered is how much they saved, and how well they invested it. Of course, my model is way oversimplified, but that does correspond to my perception of how wealth happens in real life, where my sample of friends earns around the same as I do.

---------

One thing I should note is how the "top 1%" figure of 7% is way, way off the real life figure of 38%. Why is that? Well, the main reason is probably that the model didn't consider the possibility of enterpreneurs who can occasionally create a multi-billion-dollar company out of nothing. If Bill Gates and Warren Buffett were in the model, the figure would jump substantially from 7%.

What's more surprising, I think, is that the top 10% number was so high, at 41%. I expected it to be much lower, considering that there's so much less variation here than in the real model:

1. Here, everyone had roughly the same income, between $40,000 and $60,000. Real life, on the other hand, includes sports stars, CEOs, and other people with high productivity.

2. Here, everyone was 65. Older people are obviously wealthier, since they've had much more time to earn and save. If you take these 10,000 retired people, and combine them with 10,000 babies, then the distribution is much more unequal, since you've added a bunch of zeroes. Then, the top 10% jump to from 41% of the wealth to 64%.

The age thing is a big issue. Even if everyone were exactly equal in every way, following the same career path and the same wealth accumulation path, the distribution would be unequal if you take a snapshot in time. You'd be combining 65 year olds who are rich because they've been saving, to 25 year olds who WILL be just as rich, but aren't yet. (That, by the way, is why it's best to look at lifetime income, or at least age-adjusted income, instead of snapshot income or snapshot wealth.)

Oh, and by the way ... I built a certain amount of progressive taxation into the model. I assumed all salary above $40K is taxed at 30%. I also assumed that the savings rate is based on after-tax salary. And finally, I assumed that if you save more than 30% of your after-tax salary, any excess is taxed *again* at 30%. (This was easier than trying to compute tax on investment income.) The numbers above are *after* all this progressive taxation.

----------

So, my argument boils down to something like this (directed at a random skeptic):

You say that the top 10% owns 70% of the wealth, and that's too much. Why is that too much? It can't be just inequality, because here I have a model where everyone is equal, and the top 10% still owns more than 41%. Why do you think 70% is wrong, and what should the number be, and what are your assumptions?

And suppose I cornered you, and asked you to tell me exactly what your policy prescriptions are -- how much to tax the rich, what to do with the money, how to tax investment income, what loopholes to close, how to get the poor to save more, and so on. Then I would ask you, "after all that, how much of the wealth would the top 1% own?"

I'd bet you couldn't answer that. And if you don't know what the distribution of wealth would be in your ideal world, how can you possibly argue that it's the wrong number now?

Academic editor resigns after publishing flawed study

2011-09-03T14:23:00.004-04:00

The editor of an academic journal has resigned after publishing a study that turned out to be flawed.

There's more to it than that, of course ... it's mostly an issue of political correctness, rather than a scholarly one. The study in question was by a politically incorrect author, with a politically incorrect conclusion. The paper, it turns out, was skeptical of climate change, and there are accusations that the peer reviewers who gave it their blessing were also known skeptics.

Still, it's interesting to see how the reaction pretends that's not an issue. The resigning editor, Wolfgang Wagner, wrote:

"[The paper was] "fundamentally flawed and therefore wrongly accepted by the journal ... As the case presents itself now, the [peer review] editorial team unintentionally selected three reviewers who probably share some climate sceptic notions of the authors …the problem I see with the paper by [authors] Spencer and Braswell is not that it declared a minority view (which was later unfortunately much exaggerated by the public media) but that it essentially ignored the scientific arguments of its opponents."

So, let me get the implications straight.

1. It is a very serious matter if a flawed paper is accepted by a journal.
2. If the peer reviewers agree with the author on a related scientific theory prior to the paper being published, you should find other peer reviewers.
3. If a paper ignores the scientific arguments of its opponents, it should not be published.

Can these people possibly be serious?

1. Flawed papers are accepted by journals ALL THE TIME. The sabermetric community has revealed the flaws in many, many academic studies, and no editor has resigned. Indeed, on several occasions, we have revealed problems with studies when they're still in the "working paper" stage, and they get published anyway.

If an editor had to resign every time a flawed paper got published, no editor would last longer than three months in the job.

2. This only seems to become a principle when the scientific theory in question is politically incorrect. If a journal considers a study *confirming* climate change, do they really go out of their way to find climate skeptics to peer review it? If a psychology journal publishes a study documenting the negative effects of racial bias, do they insist that one of the peer reviewers be a KKK member?

3. Actually, I'm OK with this one. But I can't resist snarping a little bit. Has any editor ever been forced to resign because of the publication of an academic paper on sabermetrics that doesn't know who Bill James is, that barely cites any existing sabermetric research, and that could have been refuted by a sabermetrician in fifteen seconds?

Okay, I'm done snarping. Moving on now.

-------

Have you ever noticed what a big deal it is in academia whenever a study is acknowledged to be flawed? The hands wring, the "mea culpa"s flow, and everyone talks about what could have gone wrong with the process that this was allowed to happen.

It looks good at first, that academia is so concerned about getting it right that they take it so seriously when something is wrong. But, really, it's a veneer, isn't it? They're just trying to signal how serious and ethical they are to people who don't know any better.

More often than not, it doesn't work that way in real life. We've all seen and talked about studies that are obviously flawed, and we've seen examples of academics who deflect the arguments to irrelevant side issues, ignore them completely, or attack our credentials instead of the actual criticism at hand.

Sometimes the community will be a bit more subtle than that ... they won't disown the study explicitly; instead, they'll publish a rebuttal letter, or a study opposing the original. They'll try to position it as a healthy scientific debate between scientists.

But, no matter how flawed the paper, they don't normally demand that the editor resign. And there also seems to be an implicit understanding that you don't pillory the original author, even if the original study was obviously meritless. You just make sure you never cite it favorably, and everyone ignores it and gets on with their lives.

But here, they won't do that. The climate change research community seems to hold the offending author in very low regard. Normally, they'd just ignore the offending author, knowing their peers would extend to them the same courtesy. But it seems like they're just fed up with this Spencer guy. And, perhaps that's for good reason. As one professor said,

"Spencer [one of the co-authors] is well known in the scientific community for publishing high-profile papers that initially dispute global warming and only later are found to be faulty."

Still, this is not a case of academia standing up to defend its strict standards of truth. This seems to be a case of academia having decided that this particular academic is persona non grata on this particular subject, and that they're not going to let him, or his editor, get away with things other academics can.

------

And you know, this turn of events might have been totally the right thing to do. I don't know this Spencer guy. For all I know, he might be an awful scientist, blinded by his political beliefs, trying to publish bad papers anywhere he can get away with, to cast doubt on the climate change hypothesis. In that case, we might agree that because of his repeated disingenuousness, and the poor quality of his work in the past, his latest study should have been subject to extra scrutiny -- and his editor's failure do that represents a resignable offense. (Again, I don't know Spencer's work at all, so this is entirely hypothetical.)

But if that's the case, say so! Don't say "the editor was fired because the paper was flawed." It makes you academics sound ridiculous, like you consider yourselves more infallible than the pope. It sends the idea that everything that makes it into a journal is invariably 100 percent correct -- because, if it weren't, would the editor still be working here?

If you say, "the editor was ~~fired~~ expected to resign because the paper was flawed," you sound arrogant and silly, not to mention dishonest.

The critics should just tell the truth. They should say, "Look, there's this one guy who's acting like a dork. He's putting together these crappy studies, which have no scientific merit, and he won't do what scientists are supposed to do and look at the data objectively. He's so politically committed to his hypothesis that he doesn't care about making his studies hold together, and he gets everything wrong.

"Now, we're scientists, so we are very open to the idea that our current theories might be wrong, and there are skeptical scientists whose research is valid and whom we respect. But not this guy. His work has been so bad, for so long, that it's incumbent on any editor to double and triple-check his work to make sure he's not doing it again. In that light, when his editor failed to do that, it's akin to negligence. So it's only appopriate that he resign."

That would make sense. But, I guess, to the general public, it doesn't look as good.

Sabermetric Research

Do NHL teams get a boost after killing a two-man advantage?

GiveWell: Overcomplicating research studies can cost lives

Are more NHL penalties called in back-to-back games?

Do hockey fights lift a team's performance? Part II

Do hockey fights lift a team's performance?

Do NHL referees call "make up" penalties? Part IV

Ken Dryden tracers from "The Game"

Do NHL referees call "make up" penalties? Part III

Do NHL referees call "make up" penalties? Part II

Do NHL referees call "make up" penalties?

The best goalies should play for the worst teams

Will taxing the rich improve democracy?

Are there "good team" goalies and "bad team" goalies?

Ken Dryden's "The Game"

A "Grantland" article on Moneyball effects

Transparent studies are better, even if they're less rigorous

Why it's hard to estimate small effects

Why p-value isn't enough, reiterated

Research conclusions *have* to be bayesian

"Statisticians can prove almost anything"

A research study is just a peer-reviewed argument

The main economic benefit of baseball: we love it

"Baseball Analyst" archives now available

A rudimentary NFL season simulation

Being proven wrong is like winning the lottery

In defense of online rudeness

How much does "Moneyball" help a team?

Capital Gains and Warren Buffett: Part II

Capital Gains and Warren Buffett: Part I

Would MLB salaries drop if all players were free agents? Part II

Would MLB salaries drop if all players were free agents?

Why Warren Buffett is wrong about tax rates

How good are sports pundits' predictions?

The Bayesian Cy Young

Stock market integrity and the OSC's bizarre Catch-22

Bob McCown on "puck luck"

Logical thinking

On inequality of wealth

Academic editor resigns after publishing flawed study

Research conclusions have to be bayesian