Skip to content

Tag: case selection

Intuitions about case selection are often wrong

Imagine the following simple setup: there are two switches (X and Z) and a lamp (Y). Both switches and the lamp are ‘On’. You want to know what switch X does, but you have only one try to manipulate the switches. Which one would you choose to switch off: X, Z or it doesn’t matter? These are the results of the quick Twitter poll I did on the question: Two switches X and Z control lamp Z. Both switches & the lamp are On. You wanna learn what X does. You have one try. Which switch to press? — Dimiter Toshkov (@DToshkov) September 4, 2017 Clearly, almost half of the respondents think it doesn’t matter, switching X is the second choice, and only 2 out of 15 would switch Z to learn what X does. Yet, it is by pressing Z that we have the best chance of learning something about the effect of X. This seems quite counter-intuitive, so let me explain. First, let’s clarify the assumptions embedded in the setup: (A1) both switches and the lamp can be either ‘On’ [1 ] or ‘Off’ [0]; (A2) the lamp is controlled only by these switches; there is nothing outside the system that controls its output; (A3) X and Z can work individually or in combination (so that the lamp is ‘On’ only if both switches are ‘On’ simultaneously). Now let’s represent the information we have in a table: Switch X Switch Z Lamp Y 1 1 1 0 0 0 We are…

Natural experiments of history? Not really, but still a fine book

Natural experiments are a fine (and fun) way to study questions where the researcher doesn’t have control over the assignment of cases. But the label ‘natural experiment’ can get abused – not all comparisons are ‘natural experiments’. Nature needs to intervene into the assignment of cases in a way that can be credibly regarded as random in order to approximate the experimental method (e.g. here). Jared Diamond and Paul Robinson have collected seven essays in a book entitled “Natural Experiments of History”. But from the seven studies, only one or two might be regarded as a true ‘natural experiment’ – the rest are just more or less systematic comparisons. It is still a fine book – I found all seven essays interesting and stimulating. But they are not natural experiments; in fact, Diamond and Robinson themselves seem to retract from the label in the concluding chapter of the book. For example, in his chapter Patrick Kirch studies Polynesian cultural evolution. The three islands of Mangaia, Marquesas, and Hawai”i end up with quite different social and political institutions despite being populated by the same people. To his credit, Kirch uses ‘controlled comparisons’ instead of ‘natural experiments’ to describe his approach. But does the fact that the three islands were settled by people coming from the same homeland in Western Polynesia allow us to ‘control’ for the pre-settlement characteristics of the people who inhabited Mangaia, Marquesas, and Hawai’i? The explorers leaving in search of new homes are seldom a representative sample from…