{"id":919,"date":"2017-09-05T12:39:54","date_gmt":"2017-09-05T12:39:54","guid":{"rendered":"http:\/\/re-design.dimiter.eu\/?p=919"},"modified":"2017-09-06T12:58:12","modified_gmt":"2017-09-06T12:58:12","slug":"intuitions-about-case-selection-are-often-wrong","status":"publish","type":"post","link":"http:\/\/re-design.dimiter.eu\/?p=919","title":{"rendered":"Intuitions about case selection are often wrong"},"content":{"rendered":"<p><strong>Imagine<\/strong> the following simple setup: there are two switches (X and Z) and a lamp (Y). Both switches and the lamp are &#8216;On&#8217;. You want to know what switch X does, but you have only one try to manipulate the switches. Which one would you choose to switch off: X, Z or it doesn&#8217;t matter?<\/p>\n<p>These are the results of the quick Twitter poll I did on the question:<\/p>\n<blockquote class=\"twitter-tweet\" data-lang=\"en\">\n<p dir=\"ltr\" lang=\"en\">Two switches X and Z control lamp Z. Both switches &amp; the lamp are On. You wanna learn what X does. You have one try. Which switch to press?<\/p>\n<p>\u2014 Dimiter Toshkov (@DToshkov) <a href=\"https:\/\/twitter.com\/DToshkov\/status\/904672045941878784\">September 4, 2017<\/a><\/p><\/blockquote>\n<p><script async src=\"\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<p>Clearly, almost half of the respondents\u00a0think it doesn&#8217;t matter, switching\u00a0X is the second choice, and only 2\u00a0out of 15 would switch Z to learn what X does. <strong>Yet, it is by pressing Z that we have the best chance of learning something about the effect of X.<\/strong> This seems quite counter-intuitive, so let me explain.<\/p>\n<p>First, let&#8217;s clarify the assumptions embedded in the setup: (A1) both switches and the lamp can be either &#8216;On&#8217; [1 ] or &#8216;Off&#8217; [0]; (A2) the lamp is controlled only by these switches; there is nothing outside the system that controls its output; (A3) X and Z can work individually or in combination (so that the lamp is &#8216;On&#8217; only if both switches are &#8216;On&#8217; simultaneously).<\/p>\n<p>Now let&#8217;s represent the information we have in a table:<\/p>\n<table style=\"width: 100%;\">\n<tbody>\n<tr>\n<th>Switch X<\/th>\n<th>Switch Z<\/th>\n<th>Lamp Y<\/th>\n<\/tr>\n<tr>\n<td>1<\/td>\n<td>1<\/td>\n<td>1<\/td>\n<\/tr>\n<tr>\n<td>0<\/td>\n<td>0<\/td>\n<td>0<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>We are allowed to make one experiment in the setup (press only one switch). In other words, we can add an observation for\u00a0one more row of the table. <strong>Which one\u00a0should it be?<\/strong><\/p>\n<p>Well, let&#8217;s see what happens if we switch off X (let&#8217;s call this strategy S1). There are two possible outcomes: either the lamp goes off (S1a) or it stays on (S1b).<\/p>\n<p>In the first case (represented as the second\u00a0line in the table below) we can conclude that X is not <em>necessary<\/em> for the lamp to be &#8216;On&#8217;, but we do not know whether X can switch on the lamp on its own (whether it is sufficient to do so).<\/p>\n<table style=\"width: 100%;\">\n<tbody>\n<tr>\n<th>Switch X<\/th>\n<th>Switch Z<\/th>\n<th>Lamp Y<\/th>\n<\/tr>\n<tr>\n<td>1<\/td>\n<td>1<\/td>\n<td>1<\/td>\n<\/tr>\n<tr>\n<td>0<\/td>\n<td>1<\/td>\n<td>1<\/td>\n<\/tr>\n<tr>\n<td>0<\/td>\n<td>0<\/td>\n<td>0<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>If\u00a0the lamp goes off when we press X, we know that X is necessary for the outcome but we do not know whether X can turn on the lamp on its own or only in combination with Z.<\/p>\n<table style=\"width: 100%;\">\n<tbody>\n<tr>\n<th>Switch X<\/th>\n<th>Switch Z<\/th>\n<th>Lamp Y<\/th>\n<\/tr>\n<tr>\n<td>1<\/td>\n<td>1<\/td>\n<td>1<\/td>\n<\/tr>\n<tr>\n<td>0<\/td>\n<td>1<\/td>\n<td>0<\/td>\n<\/tr>\n<tr>\n<td>0<\/td>\n<td>0<\/td>\n<td>0<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><strong>To sum up, by pressing X we learn either that (S1a) X is not necessary or that (S1b) X matters but we do not know whether on its own or only in combination with Z.<\/strong><\/p>\n<p>&nbsp;<\/p>\n<p>Now, let&#8217;s see what happens if we press Z (strategy S2). Again either the lamp stays on (S2a) or it goes off (S2b).<\/p>\n<p>Under the first scenario, we learn that X is sufficient to turn on the lamp.<\/p>\n<table style=\"width: 100%;\">\n<tbody>\n<tr>\n<th>Switch X<\/th>\n<th>Switch Z<\/th>\n<th>Lamp Y<\/th>\n<\/tr>\n<tr>\n<td>1<\/td>\n<td>1<\/td>\n<td>1<\/td>\n<\/tr>\n<tr>\n<td>1<\/td>\n<td>0<\/td>\n<td>1<\/td>\n<\/tr>\n<tr>\n<td>0<\/td>\n<td>0<\/td>\n<td>0<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Under the second scenario, we learn that X is not sufficient to turn on the light. It is still possible that it is necessary for turning on the lamp in combination with Z.<\/p>\n<table style=\"width: 100%;\">\n<tbody>\n<tr>\n<th>Switch X<\/th>\n<th>Switch Z<\/th>\n<th>Lamp Y<\/th>\n<\/tr>\n<tr>\n<td>1<\/td>\n<td>1<\/td>\n<td>1<\/td>\n<\/tr>\n<tr>\n<td>1<\/td>\n<td>0<\/td>\n<td>0<\/td>\n<\/tr>\n<tr>\n<td>0<\/td>\n<td>0<\/td>\n<td>0<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><strong>To sum up, by pressing Z we learn either that (S2a) X can turn on the lamp or (S2b) that it cannot turn on the lamp on its own but is possibly necessary\u00a0in combination with Z.\u00a0<\/strong><\/p>\n<p>Comparing the two sets of inferences, I think it is clear\u00a0that the second one is much more informative. By pressing Z we learn either that we can turn on the lamp by pressing X or that we cannot unless Z is &#8216;On&#8217;. By pressing X we learn next to nothing: we are either still in the dark whether X works on its own to turn on the lamp (sorry for the pun) or that X matters but we still do now know whether we\u00a0also need Z to be &#8216;On&#8217;.<\/p>\n<p>If you are still unconvinced, the following table summarizes all inferences under all strategies and contingencies about each of the possible effects (X, Z, and the interaction XZ):<\/p>\n<table style=\"width: 100%;\">\n<tbody>\n<tr>\n<th>X works on its own<\/th>\n<th>Z works on its own<\/th>\n<th>Only XZ works<\/th>\n<th>Strategy<\/th>\n<\/tr>\n<tr>\n<td>?<\/td>\n<td>True<\/td>\n<td>False<\/td>\n<th>S1a<\/th>\n<\/tr>\n<tr>\n<td>?<\/td>\n<td>False<\/td>\n<td>?<\/td>\n<th>S1b<\/th>\n<\/tr>\n<tr>\n<td>True<\/td>\n<td>?<\/td>\n<td>False<\/td>\n<th>S2a<\/th>\n<\/tr>\n<tr>\n<td>False<\/td>\n<td>?<\/td>\n<td>?<\/td>\n<th>S2b<\/th>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>It should be obvious\u00a0now that we are better off by pressing Z to learn about the effect of X.<\/p>\n<p>Good, but w<strong>hat&#8217;s the relevance<\/strong> of this little game? Well, the game resembles a research design situation in which we have one observation (case), we have the resources to add only one more, and we have to select which observation to make. In other words, the game is about case selection.<\/p>\n<p>For example, we observe a case with a rare outcome &#8211; say, successful regional integration. We suspect that two factors are at play, both of which are present in the case &#8211; say, high trade volume within the integrating block and democratic form of government for all units. And\u00a0we\u00a0wanna probe the effect of trade volume in particular. In that case, the analysis above suggests that we should choose a case that has the same volume of trade but a non-democratic form of\u00a0government, rather than a case which has low volume of trade and democratic form of government.<\/p>\n<p>This result is counter-intuitive, so let&#8217;s spell out why. First, note that we are interested in the effect of X (the effect of the switch\u00a0and of trade volume) and not in explaining Y (how to turn on the lamp\u00a0or how does regional integration come about). This is a subtle difference in interpretation, but one that is crucial for the analysis. Second, note that we are more interested in the effect of X than in the effect of Z, although both are potential causes of Y. If both X and Z are of equal interest, then obviously it doesn&#8217;t matter which one observation we make. Third, the result hinges on the assumption that there is nothing\u00a0other than X or Z (or their interaction) that matters for Y. Once we admit other possible causal variables in the set-up, then we are no longer better off switching Z to learn the effect of X.<\/p>\n<p><strong>Sooooo, don&#8217;t take this little game as general advice on case selection.<\/strong> <strong>But it definitely shows that when it comes to research design our intuitions cannot always be trusted.<\/strong><\/p>\n<p><em>P.S. One assumption on which the analysis does not depend is binary effects and outcomes: it works equally well with\u00a0probabilistic effects that are additive or multiplicative (involving an interaction).\u00a0<\/em><\/p>\n<p><a href=\"http:\/\/www.amazon.co.uk\/gp\/product\/113734282X\/ref=as_li_tl?ie=UTF8&amp;camp=1634&amp;creative=6738&amp;creativeASIN=113734282X&amp;linkCode=as2&amp;tag=rulesofreason-21\" target=\"_blank\">Learn more about research design<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Imagine the following simple setup: there are two switches (X and Z) and a lamp (Y). Both switches and the lamp are &#8216;On&#8217;. You want to know what switch X does, but you have only one try to manipulate the switches. Which one would you choose to switch off: X, Z or it doesn&#8217;t matter? These are the results of the quick Twitter poll I did on the question: Two switches X and Z control lamp Z. Both switches &amp; the lamp are On. You wanna learn what X does. You have one try. Which switch to press? \u2014 Dimiter Toshkov (@DToshkov) September 4, 2017 Clearly, almost half of the respondents\u00a0think it doesn&#8217;t matter, switching\u00a0X is the second choice, and only 2\u00a0out of 15 would switch Z to learn what X does. Yet, it is by pressing Z that we have the best chance of learning something about the effect of X. This seems quite counter-intuitive, so let me explain. First, let&#8217;s clarify the assumptions embedded in the setup: (A1) both switches and the lamp can be either &#8216;On&#8217; [1 ] or &#8216;Off&#8217; [0]; (A2) the lamp is controlled only by these switches; there is nothing outside the system that controls its output; (A3) X and Z can work individually or in combination (so that the lamp is &#8216;On&#8217; only if both switches are &#8216;On&#8217; simultaneously). Now let&#8217;s represent the information we have in a table: Switch X Switch Z Lamp Y 1 1 1 0 0 0 We are&#8230;<\/p>\n<div class=\"more-link-wrapper\"><a class=\"more-link\" href=\"http:\/\/re-design.dimiter.eu\/?p=919\">Continue reading<span class=\"screen-reader-text\">Intuitions about case selection are often wrong<\/span><\/a><\/div>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","jetpack_publicize_message":"Intuitions about case selection are often wrong","jetpack_is_tweetstorm":false},"categories":[720],"tags":[115,116,562],"jetpack_featured_media_url":"","jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p7g3hj-eP","jetpack-related-posts":[{"id":350,"url":"http:\/\/re-design.dimiter.eu\/?p=350","url_meta":{"origin":919,"position":0},"title":"David Graeber's 'Debt' will shake your world","date":"March 7, 2012","format":false,"excerpt":"David Graeber's 'Debt: The First 5,000 Year' is easily the most thought-provoking, insightful, erudite and provocative book I have read over the last few years. While you can disagree with particular arguments or resist certain conclusions, it will shake your most fundamental assumptions about social life. After reading the book,\u2026","rel":"","context":"In &quot;Anthropology&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":331,"url":"http:\/\/re-design.dimiter.eu\/?p=331","url_meta":{"origin":919,"position":1},"title":"Explanation and the quest for 'significant' relationships. Part I","date":"February 17, 2012","format":false,"excerpt":"The ultimate goal of social science is causal explanation*. The\u00a0actual goal of most\u00a0academic research is to discover significant relationships between variables. The two goals are supposed to be strongly related - by discovering (the) significant effects of exogenous (independent) variables, one\u00a0accounts for\u00a0the outcome of interest.\u00a0In fact, the working assumption of\u2026","rel":"","context":"In &quot;Causality&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":749,"url":"http:\/\/re-design.dimiter.eu\/?p=749","url_meta":{"origin":919,"position":2},"title":"Swimming in a sea of code","date":"March 28, 2013","format":false,"excerpt":"If you are looking for code here, move on. >\u00a0In the beginning, there was only the relentless blinking of the cursor. With the maddening regularity of waves splashing on the shore: blink, blink, blink, blink\u2026Beyond the cursor, the white wasteland of the empty page: vast, featureless, and terrifying as the\u2026","rel":"","context":"In &quot;R&quot;","img":{"alt_text":"","src":"https:\/\/i2.wp.com\/re-design.dimiter.eu\/wp-content\/uploads\/2013\/03\/post-9-126299637087_thumb.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":253,"url":"http:\/\/re-design.dimiter.eu\/?p=253","url_meta":{"origin":919,"position":3},"title":"Unit of analysis vs. Unit of observation","date":"January 27, 2012","format":false,"excerpt":"Having graded another batch of 40\u00a0student research proposals, the distinction between 'unit of analysis' and 'unit of observation' proves to be, yet again,\u00a0one of the trickiest for the students to master. After several years of experience, I think I have a good grasp of the difference between the two, but\u2026","rel":"","context":"In &quot;Observational studies&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":354,"url":"http:\/\/re-design.dimiter.eu\/?p=354","url_meta":{"origin":919,"position":4},"title":"Compiling government positions from the Manifesto Project data with R","date":"March 12, 2012","format":false,"excerpt":"****N.B. I have updated the functions in February 2019 to makes use of the latest Manifesto data. See for details here.*** The Manifesto Project (former Manifesto Research Group, Comparative Manifestos Project) has assembled a database of 'quantitative content analyses of parties\u2019 election programs from more than 50 countries covering all\u2026","rel":"","context":"In &quot;Policy making&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":210,"url":"http:\/\/re-design.dimiter.eu\/?p=210","url_meta":{"origin":919,"position":5},"title":"Natural experiments of history? Not really, but still a fine book","date":"December 11, 2011","format":false,"excerpt":"Natural experiments are a fine (and fun) way to study questions where the researcher doesn't have control over the assignment of cases. But the label 'natural experiment' can get abused - not all comparisons are 'natural experiments'. Nature needs to intervene into the assignment of cases in a way that\u2026","rel":"","context":"In &quot;History&quot;","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/posts\/919"}],"collection":[{"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=919"}],"version-history":[{"count":11,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/posts\/919\/revisions"}],"predecessor-version":[{"id":930,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/posts\/919\/revisions\/930"}],"wp:attachment":[{"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=919"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=919"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=919"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}