Skip to content

RE-DESIGN Posts

The Discursive Dilemma and Research Project Evaluation

Published by demetriodor on November 21, 2017

tl; dr When we collectively evaluate research proposals, we can reach the opposite verdict depending on how we aggregate the individual evaluations, and that’s a problem, and nobody seems to care or provide guidance how to proceed. Imagine that three judges need to reach a verdict together using majority rule. To do that, the judges have to decide independently if each of two factual propositions related to the suspected crime is true. (And they all agree that if and only if both propositions are true, the defendant is guilty). The distribution of the judges’ beliefs is given in the table below. Judge 1 believes that both propositions are true, and as a result, considers the conclusion (defendant is guilty) true as well. Judges 2 and 3 consider that only one of the propositions is true and, as a result, reach a conclusion of ‘not guilty’. When the judges vote in accordance with their conclusions, a majority finds the defendant ‘not guilty’.   Proposition 1 Proposition 2 Conclusion Judge 1 true true true (guilty) Judge 2 false true false (not guilty) Judge 3 true false false (not guilty) Majority decision TRUE TRUE FALSE (not guilty) However, there is a majority that finds each of the two propositions true (see the last line in the table)! Therefore, if the judges vote on each proposition separately rather than directly on the conclusion, they will have to find the defendant ‘guilty’. That is, the judges will reach the opposite conclusion, even though nothing changes about their beliefs, they still agree that both…

Continue readingThe Discursive Dilemma and Research Project Evaluation

Intuitions about case selection are often wrong

Published by demetriodor on September 5, 2017

Imagine the following simple setup: there are two switches (X and Z) and a lamp (Y). Both switches and the lamp are ‘On’. You want to know what switch X does, but you have only one try to manipulate the switches. Which one would you choose to switch off: X, Z or it doesn’t matter? These are the results of the quick Twitter poll I did on the question: Two switches X and Z control lamp Z. Both switches & the lamp are On. You wanna learn what X does. You have one try. Which switch to press? — Dimiter Toshkov (@DToshkov) September 4, 2017 Clearly, almost half of the respondents think it doesn’t matter, switching X is the second choice, and only 2 out of 15 would switch Z to learn what X does. Yet, it is by pressing Z that we have the best chance of learning something about the effect of X. This seems quite counter-intuitive, so let me explain. First, let’s clarify the assumptions embedded in the setup: (A1) both switches and the lamp can be either ‘On’ [1 ] or ‘Off’ [0]; (A2) the lamp is controlled only by these switches; there is nothing outside the system that controls its output; (A3) X and Z can work individually or in combination (so that the lamp is ‘On’ only if both switches are ‘On’ simultaneously). Now let’s represent the information we have in a table: Switch X Switch Z Lamp Y 1 1 1 0 0 0 We are…

Continue readingIntuitions about case selection are often wrong

More on QCA solution types and causal analysis

Published by demetriodor on September 4, 2017

Following up my post on QCA solution types and their appropriateness for causal analysis, Eva Thomann was kind enough to provide a reply. I am posting it here in its entirety : Why I still don’t prefer parsimonious solutions (Eva Thomann) Thank you very much, Dimiter, for issuing this blog debate and inviting me to reply. In your blog post, you outline why, absent counterevidence, you find it justified to reject applied Qualitative Comparative Analysis (QCA) paper submission that do not use the parsimonious solution. I think I agree with some but not all of your points. Let me start by clarifying a few things. Point of clarification 1: COMPASSS statement is about bad reviewer practice It´s good to see that we all seem to agree that “no single criterion in isolation should be used to reject manuscripts during anonymous peer review”. The reviewer practice addressed in the COMPASSS statement is a bad practice. Highlighting this bad reviewer practice is the sole purpose of this statement. Conversely, the COMPASSS statement does not take sides when it comes to preferring specific solution types over others. The statement also does not imply anything about the frequency of this reviewer practice – this part of your post is pure speculation.  Personally I have heard people complaining about getting papers rejected for promoting or using conservative (QCA-CS), intermediate (QCA-IS) and parsimonious solutions (QCA-PS) with about the same frequency. But it is of course impossible for COMPASSS to get a representative picture of this phenomenon. The…

Continue readingMore on QCA solution types and causal analysis

QCA solution types and causal analysis

Published by demetriodor on August 25, 2017

Qualitative Comparative Analysis (QCA) is a relative young research methodology that has been frequently under attack from all corners, often for the wrong reasons. But there is a significant controversy brewing up within the community of people using  set-theoretic methods (of which QCA is one example) as well. Recently, COMPASSS – a prominent network of scholars interested in QCA – issued a Statement on Rejecting Article Submissions because of QCA Solution Type. In this statement they ‘express the concern … about the practice of some anonymous reviewers to reject manuscripts during peer review for the sole, or primary, reason that the given study chooses one solution type over another’. The ‘solution type’ refers to the procedure used to minimize the ‘truth tables’ which collect the empirical data in QCA (and other set-theoretic) research when there are unobserved combinations of conditions (factors, variables) in the data. Essentially, in cases of missing data (which is practically always) together with the data minimization algorithm the solution type determines the inference you get from the data. I have not been involved in drawing up the statement (and I am not a member of COMPASSS), and I have not reviewed any articles using QCA recently, so I am not directly involved in this controversy on either side. At the same time, I have been interested in QCA and related methodologies for a while now, I have covered their basics in my textbook on research design, and I remain intrigued both by their promise and their…

Continue readingQCA solution types and causal analysis

Is there an East-West divide in the European Union?

Published by demetriodor on March 31, 2017

From the way the media reports on European Union negotiations, it is easy to get the impression that there is a rift between East and West European member states, and that the enlargement of the EU  has compromised the EU’s decision-making capacity. In a text published at the EUROPP blog, I argue that there is no systematic evidence to support such claims: “…in fact, against all odds, since 2004 the EU has managed to accommodate and integrate without much turbulence 13 new member states within its decision-making structures. This success is most remarkable and provides an important lesson for the future; a lesson that should not be overshadowed by the forthcoming exit of the United Kingdom from the EU.” Read the whole thing; it has pretty pictures, too.

Is interpretation descriptive or explanatory?

Published by demetriodor on February 2, 2017

One defining feature of interpretivist approaches to social science is the idea that the goal of analysis is to provide interpretations of social reality rather than law-based explanations. But of course nobody these days believes in law-based causality in the social world anyways, so the question whether interpretation is to be understood as purely descriptive or as explanatory remains. Here is what I wrote about this issue for an introductory chapter on research design in political science. The paragraph, however, will need to be removed from the text to make the chapter shorter, so I post it here instead. I will be glad to see opinions from scholars who actually work with interpretivist methodologies: It is difficult to position interpretation (in the narrow sense of the type of work interpretivist political scientists engage in) between description and explanation. Clifford Geertz notes that (ethnographic) description is interpretive (Geertz 1973: 20), but that still leaves the question whether all interpretation is descriptive open. Bevir and Rhodes (2016) insist that intepretivists reject a ‘scientific concept of causation’, but suggest that we can explain actions as products of subjective reasons, meanings, and beliefs. In addition, intentionalist explanations are to be supported by ‘narrative explanations’. In my view, however, a ‘narrative’ that ‘explains’ by relating actions to beliefs situated in a historical context is conceptually and observationally indistinguishable from a ‘thick description’, and better regarded as such.