QCA solution types and causal analysis

Last updated on September 13, 2017

Qualitative Comparative Analysis (QCA) is a relative young research methodology that has been frequently under attack from all corners, often for the wrong reasons. But there is a significant controversy brewing up within the community of people using  set-theoretic methods (of which QCA is one example) as well.

Recently, COMPASSS – a prominent network of scholars interested in QCA – issued a Statement on Rejecting Article Submissions because of QCA Solution Type. In this statement they ‘express the concern … about the practice of some anonymous reviewers to reject manuscripts during peer review for the sole, or primary, reason that the given study chooses one solution type over another’. The ‘solution type’ refers to the procedure used to minimize the ‘truth tables’ which collect the empirical data in QCA (and other set-theoretic) research when there are unobserved combinations of conditions (factors, variables) in the data. Essentially, in cases of missing data (which is practically always) together with the data minimization algorithm the solution type determines the inference you get from the data.

I have not been involved in drawing up the statement (and I am not a member of COMPASSS), and I have not reviewed any articles using QCA recently, so I am not directly involved in this controversy on either side. At the same time, I have been interested in QCA and related methodologies for a while now, I have covered their basics in my textbook on research design, and I remain intrigued both by their promise and their limitations. So I feel like sharing my thoughts on the matter, even if others might have much more experience with QCA.

(1a) First, let me say that no matter what one thinks about the appropriateness of a solution type, no single criterion in isolation should be used to reject manuscripts during anonymous peer review. The reviewer’s recommendation should reflect not only the methodology used, but the original research goal and the types of inferences being made. I can only assume that for COMPASSS to issue such a statement, the problem has been one of systematic rejection due to this one single reason. This is worrisome because the peer review process does not offer possibilities for response, let alone for debate of methodological issues.

(1b) At the same time, if the method that is used is not appropriate for the research goal and does not support the inferences advanced in the manuscript, then rejection is warranted and no further justification is needed.

(2a) So it all depends whether a single solution type should be used in all QCA analyses. In principle, the answer to this question is ‘No’. There are three main types of solutions (parsimonious, complex, and intermediate), and each can be appropriate in different circumstances.

(2b) However, when it comes to causal analysis, my answer is ‘Yes. Only the parsimonious solution should be used to make causal inferences.’ My answer is based on Michael Baumgartner’s analysis (see the 2015 version here), and I will explain why I find it persuasive below. So, if manuscripts make causal claims based on non-parsimonious solution types, I would see that as sufficient grounds for rejection (or rather for revision), unless the authors explicitly subscribe to a very peculiar social ontology in which everything has causal relevance for an outcome unless we have evidence to the contrary (I will explain this below). In my view, the standard ontology is that no factor has causal relevance for an outcome unless we have evidence that it does.

To sum up so far, the COMPASSS people might be right in general, but for the important class of causal analyses they are wrong.  

(3) Why only the parsimonious solution should be used to make causal claims? In short, because the relations of necessity and sufficiency are monotonic (so that if A is sufficient for E, AB is also sufficient, for any arbitrary B). Imagine a causal structure in which the presence of A is necessary and sufficient for the presence of E and B is irrelevant. Further imagine that we only have two empirical observations {ABE} and {aBe} (small letters denotes the absence of the condition/outcome).  This data is incomplete as we have no information on what happens under the logically possible situations {Ab} and {ab} (these would be logical remainders in the truth table), so we have to use some further rules (e.g. solution type) to derive the outcome. The complex solution is AB->E (the presence of both A and B is necessary and sufficient for the outcome E to occur). This solution type assumes that we cannot ignore B: as we have no data on what happens when it is not present, it is prudent to assume that B matters and keep it in the resulting formula (e.g. causal recipe). However, this formula and the conclusion it leads to are wrong because we posited above that B is irrelevant. The parsimonious solution is A->E (the presence of A is necessary and sufficient for the presence of E). The solution has eliminated B making the assumption that B does not matter as this would result in a more parsimonious solution. This is the correct inference in our example.

Our example is simple but in no way contrived (Baumgartner has other, more complex examples in the paper). In fact, we can add any number of factors to A and as long as they do not vary across our cases they will appear to be components of the outcome formula (causal recipe). That is, any random factor can be made to appear as causally relevant in the presence of limited diversity in the cases being studied. In the limit, we can make every aspect of a case appear to be causally relevant for an outcome, if we do not have cases combining factors in a way that makes it possible to disprove their illusory relevance.

You can say that this is only fair. But it would only be appropriate in a world where everything matters for everything else, unless some empirical cases point to the opposite. But such a worldview (ontology) is rare among social scientists (and I have never seen it openly endorsed). Note that the problem is not that we imply a separate independent effect of B on E: worse, the solution implies that B must be present for the effect of A to obtain.

To sum up so far, only the parsimonious solution type can provide causal inference from QCA data under a standard social ontology, because of the monotonicity of the relationships of necessity and sufficiency.

(4) So what is the response of the COMPASSS people to this analysis? In fact, I do not know. To the best of  my knowledge, there has been no published response/critique to Michael Baumgartner’s article. In the Statement, the following arguments are given:

(a) ‘The field of QCA and set-theoretic methods is not quite standardized.’ ‘The field is currently witnessing an ongoing and welcome methodological debate about the correctness of different solution types.’ ‘the current state of the art is characterized by discussions between leading methodologists about these questions’.

All this might be the case, but the point about non-parsimonious solutions deriving faulty causal inferences seems settled, at least until there is a published response that rebukes it. There might be debate, but I have not seen a published response to Baumgartner’s analysis or any other persuasive argument why he is wrong on this point in particular.

(b) ‘users applying [QCA and other set-theoretic] methods who refer to established protocols of good practice must not be made responsible for the fact that, currently, several protocols are being promoted’. Well, users cannot be made responsible, but if the protocols they follow are faulty, their manuscripts cannot be accepted as the analyses would be wrong.

(5) So despite offering no reasons why non-parsimonious solutions are appropriate for causal analysis contra Baumgartner, COMPASSS’ statement finishes with ‘all solutions are empirically valid’.

I am not sure what this means. All solutions cannot be empirically valid as they can point to contradictory conclusions. Either A->E or AB->E; either B is causally relevant or not. Technically, any solution might be valid in light of a set of background assumptions, research goals and analytic procedures (in the sense that both 2+2=4 and 2+2=5 are valid under some assumptions.) But that’s the crux of the matter: if one has causal goals and uses the non-parsimonious solution, then the solution is only valid if one assumes that in the social world everything causes and conditions everything else unless proven otherwise.

To conclude, if a group of researchers have been systematically sabotaging the work of other scholars for the sole fact of using a certain type of solution concept, that’s bad.  But if they have been rejecting manuscripts that have used non-parsimonious solutions to derive causal inferences without clear commitments to an ‘everything matters’ worldview, that seems OK to me, in light of the (published) methodological state of the art.

P.S. The issue of counterfactuals enters this debate quite often.
(a) But in his 2015 analysis Baumgartner does not evoke his/a regularity theory of causality. All he needs for the analysis is a notion of a cause as a difference-maker, which in my understanding is compatible with a counterfactual understanding of causality. So any rejection of his argument against non-parsimonious solutions cannot be derived from differences between regularity and counterfactual notions of causality.
(b) Baumgartner notes that the parsimonious solution sometimes requires one to make counterfactuals about impossible states of the world. With this critique he motivates abandoning the Quine-McCluskey Boolean minimization procedure (in the framework of which one must choose the parsimonious, complex, or intermediate solution types) altogether and adopting the coincidence analysis framework, which has the parsimonious solution ‘built-in’ in its algorithms. But his is not a critique against counterfactuals as such.
(c) At the same time, the complex solution also relies on assumption-based counterfactuals, namely that a factor matters unless shown otherwise. So the reliance on counterfactuals cannot be used to arbitrate this debate.

For further discussion of these issues, see Thiem and Baumgarnter 2016, Ingo Rohlfing’s blog post (with a response in the comments by Michael Baumgartner), Schneider 2016, the Standards of Good Practice in QCA,

[addendum 31/08/2017] Michael Baumgartner and Alrik Thiem have published a reply to the COMPASSS statement in which they write: ‘We endorse the prerogative of journal editors and reviewers to favor rejection if they come to the conclusion that a manuscript does not merit publication because of its choice of an unsuitable solution type.’ And they urge for more debate.

Want to learn more about Qualitative Comparative Analysis? Start with any of these: Rihoux and Ragin (2008),  Schneider and Wagemann (2012), Thiem and Dusa (2012), Ragin (2000), Ragin (2014).