Intuitions about case selection are often wrong

Imagine the following simple setup: there are two switches (X and Z) and a lamp (Y). Both switches and the lamp are ‘On’. You want to know what switch X does, but you have only one try to manipulate the switches. Which one would you choose to switch off: X, Z or it doesn’t matter?

These are the results of the quick Twitter poll I did on the question:

Clearly, almost half of the respondents think it doesn’t matter, switching X is the second choice, and only 2 out of 15 would switch Z to learn what X does. Yet, it is by pressing Z that we have the best chance of learning something about the effect of X. This seems quite counter-intuitive, so let me explain.

First, let’s clarify the assumptions embedded in the setup: (A1) both switches and the lamp can be either ‘On’ [1 ] or ‘Off’ [0]; (A2) the lamp is controlled only by these switches; there is nothing outside the system that controls its output; (A3) X and Z can work individually or in combination (so that the lamp is ‘On’ only if both switches are ‘On’ simultaneously).

Now let’s represent the information we have in a table:

Switch X Switch Z Lamp Y
1 1 1
0 0 0

We are allowed to make one experiment in the setup (press only one switch). In other words, we can add an observation for one more row of the table. Which one should it be?

Well, let’s see what happens if we switch off X (let’s call this strategy S1). There are two possible outcomes: either the lamp goes off (S1a) or it stays on (S1b).

In the first case (represented as the second line in the table below) we can conclude that X is not necessary for the lamp to be ‘On’, but we do not know whether X can switch on the lamp on its own (whether it is sufficient to do so).

Switch X Switch Z Lamp Y
1 1 1
0 1 1
0 0 0

If the lamp goes off when we press X, we know that X is necessary for the outcome but we do not know whether X can turn on the lamp on its own or only in combination with Z.

Switch X Switch Z Lamp Y
1 1 1
0 1 0
0 0 0

To sum up, by pressing X we learn either that (S1a) X is not necessary or that (S1b) X matters but we do not know whether on its own or only in combination with Z.

 

Now, let’s see what happens if we press Z (strategy S2). Again either the lamp stays on (S2a) or it goes off (S2b).

Under the first scenario, we learn that X is sufficient to turn on the lamp.

Switch X Switch Z Lamp Y
1 1 1
1 0 1
0 0 0

Under the second scenario, we learn that X is not sufficient to turn on the light. It is still possible that it is necessary for turning on the lamp in combination with Z.

Switch X Switch Z Lamp Y
1 1 1
1 0 0
0 0 0

To sum up, by pressing Z we learn either that (S2a) X can turn on the lamp or (S2b) that it cannot turn on the lamp on its own but is possibly necessary in combination with Z. 

Comparing the two sets of inferences, I think it is clear that the second one is much more informative. By pressing Z we learn either that we can turn on the lamp by pressing X or that we cannot unless Z is ‘On’. By pressing X we learn next to nothing: we are either still in the dark whether X works on its own to turn on the lamp (sorry for the pun) or that X matters but we still do now know whether we also need Z to be ‘On’.

If you are still unconvinced, the following table summarizes all inferences under all strategies and contingencies about each of the possible effects (X, Z, and the interaction XZ):

X works on its own Z works on its own Only XZ works Strategy
? True False S1a
? False ? S1b
True ? False S2a
False ? ? S2b

It should be obvious now that we are better off by pressing Z to learn about the effect of X.

Good, but what’s the relevance of this little game? Well, the game resembles a research design situation in which we have one observation (case), we have the resources to add only one more, and we have to select which observation to make. In other words, the game is about case selection.

For example, we observe a case with a rare outcome – say, successful regional integration. We suspect that two factors are at play, both of which are present in the case – say, high trade volume within the integrating block and democratic form of government for all units. And we wanna probe the effect of trade volume in particular. In that case, the analysis above suggests that we should choose a case that has the same volume of trade but a non-democratic form of government, rather than a case which has low volume of trade and democratic form of government.

This result is counter-intuitive, so let’s spell out why. First, note that we are interested in the effect of X (the effect of the switch and of trade volume) and not in explaining Y (how to turn on the lamp or how does regional integration come about). This is a subtle difference in interpretation, but one that is crucial for the analysis. Second, note that we are more interested in the effect of X than in the effect of Z, although both are potential causes of Y. If both X and Z are of equal interest, then obviously it doesn’t matter which one observation we make. Third, the result hinges on the assumption that there is nothing other than X or Z (or their interaction) that matters for Y. Once we admit other possible causal variables in the set-up, then we are no longer better off switching Z to learn the effect of X.

Sooooo, don’t take this little game as general advice on case selection. But it definitely shows that when it comes to research design our intuitions cannot always be trusted.

P.S. One assumption on which the analysis does not depend is binary effects and outcomes: it works equally well with probabilistic effects that are additive or multiplicative (involving an interaction). 

Learn more about research design.

Natural experiments of history? Not really, but still a fine book

Natural experiments are a fine (and fun) way to study questions where the researcher doesn’t have control over the assignment of cases. But the label ‘natural experiment’ can get abused – not all comparisons are ‘natural experiments’. Nature needs to intervene into the assignment of cases in a way that can be credibly regarded as random in order to approximate the experimental method (e.g. here).

Jared Diamond and Paul Robinson have collected seven essays in a book entitled “Natural Experiments of History”. But from the seven studies, only one or two might be regarded as a true ‘natural experiment’ – the rest are just more or less systematic comparisons. It is still a fine book – I found all seven essays interesting and stimulating. But they are not natural experiments; in fact, Diamond and Robinson themselves seem to retract from the label in the concluding chapter of the book.

For example, in his chapter Patrick Kirch studies Polynesian cultural evolution. The three islands of Mangaia, Marquesas, and Hawai”i end up with quite different social and political institutions despite being populated by the same people. To his credit, Kirch uses ‘controlled comparisons’ instead of ‘natural experiments’ to describe his approach. But does the fact that the three islands were settled by people coming from the same homeland in Western Polynesia allow us to ‘control’ for the pre-settlement characteristics of the people who inhabited Mangaia, Marquesas, and Hawai’i? The explorers leaving in search of new homes are seldom a representative sample from the original population. They could be the most adventurous, skilled, risk-taking individuals who depart in search of better fortunes. Or they could be the most aggressive, rebellious, conflict-seeking and uncooperative individuals who get expelled from their original island. In any case, they are likely to be different than the median individual of the source population. As a result, we cannot really rule out that Mangaia, Marquesas, and Hawai’i develop different and progressively more complex political organizations because of differences between the original inhabitants rather that the ecological variables discussed by Kirch. My aim here is not to criticize the substance of Kirch’s research – I actually find his conclusions rather compelling, but to illustrate how self-selection can undermine the comparisons when the assignment of people to islands is not entirely random.

In fact, from the seven chapters the comparison of different Polynesian islands is one of the more persuasive ones from a methodological point of view. Only Banhijit Banerjee and Lakshmi Iyer’s essay on the impact of land tenure institutions on public goods provision in India comes closer to the use of a selection mechanism that can be regarded as a natural experiment. But even there, if the British governors who opted for village-based ownership were also more likely to engage in other ‘progressive’ projects (thus establishing the initial infrastructure for public good provision), then the differences we observe today between Indian regions might not be attributable to the different land tenure regimes during colonial rule.

Even if not based on true natural experiments, the seven essays collected in this little book have a lot to offer. For historians especially, some of the methodological approaches might be eye-opening and hopefully lead to more rigorous search for patterns in history.  If the price for bringing methodological insight to a bigger and more diverse audience is a catchy title that is not entirely fair to the content of the book, so be it.