{"id":915,"date":"2017-09-04T09:07:43","date_gmt":"2017-09-04T09:07:43","guid":{"rendered":"http:\/\/re-design.dimiter.eu\/?p=915"},"modified":"2017-09-04T09:10:52","modified_gmt":"2017-09-04T09:10:52","slug":"more-on-qca-solution-types-and-causal-analysis","status":"publish","type":"post","link":"http:\/\/re-design.dimiter.eu\/?p=915","title":{"rendered":"More on QCA solution types and causal analysis"},"content":{"rendered":"

Following up my post on QCA\u00a0solution types and their appropriateness for causal analysis, Eva Thomann<\/a> was kind enough to provide a reply. I am posting it here in its entirety :<\/p>\n

Why I still don’t prefer parsimonious solutions (Eva Thomann)<\/strong><\/h2>\n

Thank you very much, Dimiter, for issuing this blog debate<\/a> and inviting me to reply. In your blog post, you outline why, absent counterevidence, you find it justified to reject applied Qualitative Comparative Analysis (QCA) paper submission that do not use the parsimonious solution. I think I agree with some but not all of your points. Let me start by clarifying a few things.<\/p>\n

Point of clarification 1: COMPASSS statement is about bad reviewer practice<\/strong><\/p>\n

It\u00b4s good to see that we all seem to agree that \u201cno single criterion in isolation should be used to reject manuscripts during anonymous peer review\u201d. The reviewer practice addressed in the COMPASSS statement<\/a> is a bad practice. Highlighting this bad reviewer practice is the sole purpose of this statement.<\/em> Conversely, the COMPASSS statement does not take sides when it comes to preferring specific solution types over others. The statement also does not imply anything about the frequency of this reviewer practice \u2013 this part of your post is pure speculation.\u00a0 Personally I have heard people complaining about getting papers rejected for promoting or using conservative (QCA-CS), intermediate (QCA-IS) and parsimonious solutions (QCA-PS) with about the same frequency. But it is of course impossible for COMPASSS to get a representative picture of this phenomenon.<\/p>\n

The term \u201cempirically valid\u201d refers to the, to my best knowledge entirely undisputed fact that all solution types are (at least) based on the information contained in the empirical data. The question that\u00b4s disputed is how we can or should go \u201cbeyond the facts\u201d in causally valid ways when deriving QCA solutions.<\/p>\n

Having said this, I will take off my \u201chat\u201d as a member of the COMPASSS steering committee and contribute a few points to this debate. These points represent my own personal view and not that of COMPASSS or any of its bodies. I write as someone who uses QCA sometimes in her research and teaches it, too. Since I am not a methodologist, I won\u00b4t talk about fundamental issues of ontology and causality. I hope others will jump in on that.<\/p>\n

Point of clarification 2: There is no point in personalizing this debate<\/strong><\/p>\n

In your comment you frequently refer to \u201cthe COMPASSS people\u201d. But I find that pointless: COMPASSS<\/a> hosts a broad variety of methodologists, users, practitioners, developers and teachers with different viewpoints and of different \u201ccolours and shapes\u201d, some persons closer to \u201ccase-based\u201d research, other closer to statistical\/analytical research. Amongst others, Michael Baumgartner whom you mention is himself a members of the advisory board and he has had methodological debates with his co-authors<\/a> as well. \u00a0Just because we can procedurally agree on a bad reviewer practice, it neither means we substantively agree on everything, nor does it imply that we disagree. History has amply shown<\/a> how unproductive it can be for scientific progress when debates like these become personalized. Thus, if I could make a wish to you and everyone else engaging in this debate, it would be to talk about arguments rather than specific people<\/em>. In what follows I will therefore refer to different approaches instead unless when referring to specific scholarly publications.<\/p>\n

Point of clarification 3: There is more than one perspective on the validity of different solutions<\/strong><\/p>\n

As to your earlier point<\/a> which you essentially repeat here, that \u201cbut if two solutions produce different causal recipes,\u00a0 e.g. (1) AB-> E and (2) ABC-> E it cannot be that both (1) and (2) are valid\u201d, my answer is: it depends on what you mean with \u201cvalid\u201d.<\/p>\n

It is common to look at QCA results as subset relations<\/em>, here: statements of sufficiency. In a paper that is forthcoming in Sociological Methods & Research<\/em><\/a>, Martino Maggetti and I call this the\u201d approach emphasizing substantive interpretability\u201d. From this perspective, the forward arrow \u201c->2 reads \u201cis sufficient for\u201d and 1) in fact implies 2). Sufficiency means that X (here: AB) is a subset of Y (here: E). ABC is a subset of AB and hence it is also a subset of E, if AB is a subset of E. Logic dictates that any subset of a sufficient condition is also sufficient. Both are valid \u2013 they describe the sufficiency patterns in the data (and sometimes, some remainders) with different degrees of complexity.<\/p>\n

Scholars promoting an \u201capproach emphasizing redundancy-free models\u201d agree with that, if we speak of mere (monotonic) subset relations. Yet they require QCA solutions to be minimal statements of causal relevance<\/em>. From this perspective, the arrow (it then is <->, see below) reads \u201cis causally relevant for\u201d and if 1) then 2) cannot be true: 2) additionally grants causal relevance to C, but in in 1) we said only AB are causally relevant. As a causal statement, we can think of 2) claiming more than 1).<\/p>\n

To proponents of the approach emphasizing substantive interpretability (and I am one of them), it all boils down to the question:<\/p>\n

\u201cCan something be incorrect<\/strong> that follows logically and inevitably from a correct <\/strong>statement?<\/em> \u201c<\/p>\n

Their brains shout:<\/p>\n

\u201cNo, of course it can\u2019t!<\/em> \u201c<\/p>\n

I am making an informed guess here: this fact is so blatantly obvious to most people well-versed in set theory that it does not require a formal reply.<\/p>\n

For everyone else, it is important to understand that in order to follow the reasoning you are proposing in your comment, you have to buy into a whole set of assumptions that underlie the method promoted in the publication you are referring to (Baumgartner 2015<\/a>), called Coincidence Analysis or CNA. Let me illustrate this.<\/p>\n

Point of clarification 4: QCA is not CNA<\/strong><\/p>\n

In fact, one cannot accept 2) if 1) is true in the special case when the condition \u201cAB\u201d is \u00a0both<\/em> minimally sufficient and <\/em>contained in a minimally necessary condition for an outcome \u2013 which is also the situation you refer to (in your point 3). We have to replace the forward arrow \u201c->\u201d with \u201c<->\u201d.In such a situation, the X set and the Y set are equivalent. Of course, if AB and E are equivalent, then ABC and E are not equivalent at the same time. In reality, this \u2013 simultaneous necessity and sufficiency\u2013 is a rare scenario that requires a solution to be maximally parsimonious and having both a high consistency (indicating sufficiency) AND a very high coverage (indicating necessity).<\/p>\n

But QCA \u2013 as opposed to CNA \u2013 is designed to assess necessary conditions and \/ or sufficient conditions. They don\u00b4t have to be both. As soon as we are speaking of a condition that is sufficient but not necessary (or not part of a necessary condition), then, if 1) is correct, 2) also has to be correct. You are acknowledging this when saying that \u201cif A is sufficient for E, AB is also sufficient, for any arbitrary B\u201d.<\/p>\n

I will leave it to the methodologists to clarify whether it is ontologically desirable to empirically analyse sufficient but not necessary (or necessary but not sufficient) conditions. As a political scientist, I find it theoretically and empirically interesting. I believe this is in the tradition of much comparative political research. It is clear, and you seem to agree, that what we find to be correct entirely depends on how we define \u201ccorrect\u201d<\/a> \u2013 there\u00b4s a danger of circularity here. At this point in time, it has to be pointed out that CNA is not QCA. Both are innovative, elegant and intriguing methods with their own pro\u2019s and con\u2019s. I am personally quite fascinated by CNA and would like to see more applications of it, but I am not convinced that we can or need to transfer its assumptions to QCA.<\/p>\n

What I like about the recent publications advocating an approach emphasizing redundancy-free models is that they highlight that not all conditions contained in QCA solutions may be causally interpretable, if only we knew the true data-generating process (DGP). That points to the general question of causal arguments made with QCA if there is limited diversity, which has received ample scholarly attention for already quite a while.<\/p>\n

Point of agreement 1: We need a cumulative process of rational critique<\/strong><\/p>\n

You argue that \u201cthe point about non-parsimonious solutions deriving faulty causal inferences seems settled, at least until there is a published response that rebukes it\u201d. But QCA scholars have long highlighted issues of implausible and untenable counterfactuals entailed in parsimonious solutions (e.g. here<\/a>, here<\/a>, here<\/a>, here<\/a>, here<\/a>, here<\/a> and here<\/a>). None of the published articles advocating redundancy-free models has so far made concrete attempts to rebuke these arguments. Following your line of reasoning, the points made by previous scholarship about parsimonious solutions deriving faulty causal inferences equally seems settled, at least until there is a published response that rebukes these points.<\/p>\n

Indeed, advocates of redundancy-free models seem to either dismiss the relevance of counterfactuals altogether because CNA, so it is argued, does not rely on counterfactuals to derive solutions; OR they argue, that in the presence of limited diversity all solutions rely on counterfactuals. (Wouldn\u00b4t it be contradictory to argue both?). I personally would agree with the latter point. There can be no doubt that QCA (as opposed, perhaps, to CNA) is a set-theoretic, truth table based methods that, in the presence of limited diversity, involves counterfactuals.<\/em> Newer algorithms (such as eQMC<\/a>, used in the QCA package for R<\/a>) no longer actively \u201crely on\u201d remainders for minimization, and they exclude difficult and untenable counterfactuals rather than including tenable and \u201ceasy\u201d counterfactuals. But the reason why QCA involves counterfactuals keeps being that intermediate and parsimonious QCA solutions involve configurations of conditions some of which are empirically observed, while others (the counterfactuals) are not. There can be only one conclusion: that the question of whether these counterfactuals are valid requires our keen attention.<\/p>\n

Where does that leave us? To me, all that does certainly not mean that \u201cthe reliance on counterfactuals cannot be used to arbitrate this debate\u201d. It means that different scholars have highlighted different issues relating to the validity of all solution types. None of these points have been conclusively rebuked so far<\/em>. That, of course, leaves users in an intricate situation. They should not be punished for consistently and correctly following protocols proposed by methodologists of one or another approach.<\/p>\n

Point of agreement 2: In the presence of limited diversity, QCA solutions can err in different directions<\/strong><\/p>\n

Parsimonious solutions are by no means unaffected by the problem that limited empirical diversity challenges our confidence in inferences. Indeed we should be careful not to omit that they err, too. As Ingo Rohlfing has pointed out<\/a>, the question in which direction we want to err is a different question than the one which solution is correct. The answer to this former question probably depends.<\/p>\n

Let us return to the above example and assume that we have a truth table characterized by limited diversity. We get a conservative solution<\/p>\n

(CS) ABC -> E,<\/p>\n

and a parsimonious solution<\/p>\n

(PS) A -> E.<\/p>\n

Let us further assume that we know (which in reality we never do) that the true DGP is<\/p>\n

(DGP) AB -> E.<\/p>\n

Neither CS nor PS give us the true DGP. To recap: To scholars emphasizing redundancy-free models, PS is \u201ccorrect\u201d because they define as \u201ccorrect\u201d a solution that does not contain causally irrelevant conditions. But note that PS here is also incomplete<\/em>: the true result in this example is that, in order to observe the outcome E, A alone is not enough, it has to combine with B. Claiming that A alone is enough involves a counterfactual that could well be untenable. But the evidence alone does not allow us to conclude that B is irrelevant for E. It is usually only by making this type of oversimplifications that parsimonious solutions reach the high coverage values required to be \u201ccausally interpretable\u201d under an approach emphasizing redundancy-free models.<\/p>\n

To anyone with some basic training in QCA, this should raise some serious questions: But isn\u00b4t one of the core assumptions of QCA that we cannot interpret the single conditions in its results in isolation because they unfold their effect only in combination with other conditions?<\/em> How, then, does QCA-PS fare when assessed against this assumption? I have not read a conclusive answer to this question yet.<\/p>\n

Baumgartner and Thiem (2017)<\/a> point out that with imperfect data, no method can be expected to deliver complete results. That may well be, but in QCA we deal with two types of completeness: complete AND-configurations, or including all substitutable paths or \u201ccausal recipes\u201d combined with the logical OR. In order to interpret a QCA solution as a sufficient condition, I want to be reasonably sure that the respective AND-configuration in fact reliably implies the outcome (even if it omits other configurations that may not have been observed in my data).\u00a0 Using this criterion, QCA-PS arguably fares worst (it most often misses out on causally relevant factors) and QCA-CS fares best (though it most often also still includes causally irrelevant factors).<\/p>\n

To be sure, QCA-PS is <\/em>sufficient for the outcome in the dataset under question. But I am unsure how I have to read it: \u201ceither X implies Y, or I did not observe X\u201d? Or \u201cX is causally relevant for Y in the data under question, but I don\u00b4t know if it suffices on its own\u201d? There may well be specific situations in which all we want to know if some conditions are causally relevant subsets of sufficient conditions<\/em> or not. But I find it misleading to claim that this is the only legitimate or even the main research interest of studies using QCA. I can think of many situations, such as public health crises or enforcing EU law, in which reliably achieving or preventing an outcome would have priority.<\/p>\n

Let me be clear. The problem we are talking about is really neither QCA nor some solution type. The elephant in the room is essentially that observational data are rarely perfect and do not obey to the laws of logic.<\/em> But is QCA-PS really the best, or the only, or at all, a way out of this problem?<\/p>\n

Point of agreement 3: There are promising and less promising strategies for causal assessment <\/strong><\/p>\n

The technical moment of QCA shares with statistical techniques that it is simply a cross-case comparison of data-set observations. As such, of course it also shares with other methods the limited possibility for directly deriving causal inferences from observational data. Most QCA scholars would therefore be very cautious to interpret QCA results causally when using observational data and in the presence of limited diversity. Obviously, set relation does not equal causation<\/em>. How then, could a specific minimization algorithm alone plausibly facilitate causal interpretability?<\/p>\n

QCA (as opposed to CNA) was always designed to be a multi-method approach. This means that the inferences of the cross-case comparison are not just interpreted as such, but strengthened and complemented with additional insights, usually theoretical, conceptual and case knowledge. Or, as Ragin (2008: 173)<\/a> puts it:<\/p>\n

\u201cSocial research (\u2026) is built upon a foundation of substantive and theoretical knowledge, not just methodological technique\u201d.<\/p>\n

This way, we can combine the advantages offered by different methods and sources<\/a>. Used in a formalized way, the combination of QCA with process tracing<\/a> can even help to disentangle causally relevant from causally irrelevant conditions<\/a>. This, of course, does not preclude the possibility that some solution types may lend themselves more to causal interpretation than others. It does suggest, though, that focusing on specific solution types alone is an ill-suited strategy for making valid causal assessments.<\/p>\n

Point of disagreement: Nobody assumes that \u201ceverything matters\u201d<\/strong><\/p>\n

Allow me to disagree that an approach emphasizing substantive interpretability assumes \u201ceverything is relevant\u201d. Of course that is nonsense. Like with any other social science method I know, the researcher first screens the literature and field in order to identify potentially relevant explanatory factors. The logic of truth table analysis (as opposed to CNA?) is then to start out with the subset of these previously identified conditions that themselves consistently are a subset of the outcome set, and then it searches for evidence that they are irrelevant. This is not even an assumption, and it is very far from being \u201ceverything\u201d.<\/p>\n

Ways ahead<\/strong><\/p>\n

In my view it makes sense to have a division of labour: Users follow protocols, methodologists foster methodological innovation and progress. I hope the above has made it clear that we are in the midst of, in my view, welcome and needed debate about what \u201ccorrectness\u201d and validity\u201d means in the QCA context. I find it useful to think of this as a diversity of approaches to QCA. It is important that researchers reflect about the ontology that underlies their work, but we should avoid making premature conclusions as well.<\/p>\n

Currently (but I may be proven wrong) I am thinking that each solution type has its merits and limitations. We can\u2019t eliminate limited diversity, but we can use different solution typos for different purposes. For example, if policymakers seek to avoid investing public money in potentially irrelevant measures, the PS could be best. If they are interested in creating situations that are 100% sure to ensure an outcome (e.g. disease prevention), then the conservative solution is best and the parsimonious solution very risky. If we have strong theoretical knowledge or prior evidence available for counterfactual reasoning, intermediate solutions are best. And so on. From this perspective, it is good that we can refer to different solution types with QCA. It forces researchers to think consciously about what the goal of their analysis is, and how it can be adequately reached. It prevents them from just mechanically running some algorithm on their data.<\/p>\n

All of the above is why I agree with the COMPASSS statement that \u2026<\/p>\n

\u201cthe current state of the art is characterized by discussions between leading methodologists about these questions, rather than by definitive and conclusive answers. It is therefore premature to conclude that one solution type can generally be accepted or rejected as \u201ccorrect\u201d, as opposed to other solution types\u201d.<\/p>\n","protected":false},"excerpt":{"rendered":"

Following up my post on QCA\u00a0solution types and their appropriateness for causal analysis, Eva Thomann was kind enough to provide a reply. I am posting it here in its entirety : Why I still don’t prefer parsimonious solutions (Eva Thomann) Thank you very much, Dimiter, for issuing this blog debate and inviting me to reply. In your blog post, you outline why, absent counterevidence, you find it justified to reject applied Qualitative Comparative Analysis (QCA) paper submission that do not use the parsimonious solution. I think I agree with some but not all of your points. Let me start by clarifying a few things. Point of clarification 1: COMPASSS statement is about bad reviewer practice It\u00b4s good to see that we all seem to agree that \u201cno single criterion in isolation should be used to reject manuscripts during anonymous peer review\u201d. The reviewer practice addressed in the COMPASSS statement is a bad practice. Highlighting this bad reviewer practice is the sole purpose of this statement. Conversely, the COMPASSS statement does not take sides when it comes to preferring specific solution types over others. The statement also does not imply anything about the frequency of this reviewer practice \u2013 this part of your post is pure speculation.\u00a0 Personally I have heard people complaining about getting papers rejected for promoting or using conservative (QCA-CS), intermediate (QCA-IS) and parsimonious solutions (QCA-PS) with about the same frequency. But it is of course impossible for COMPASSS to get a representative picture of this phenomenon. The…<\/p>\n

Continue readingMore on QCA solution types and causal analysis<\/span><\/a><\/div>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false},"categories":[8,720],"tags":[121,722,721],"jetpack_featured_media_url":"","jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p7g3hj-eL","jetpack-related-posts":[{"id":905,"url":"http:\/\/re-design.dimiter.eu\/?p=905","url_meta":{"origin":915,"position":0},"title":"QCA solution types and causal analysis","date":"August 25, 2017","format":false,"excerpt":"Qualitative Comparative Analysis (QCA) is a relative young research methodology that has been frequently under attack from all corners, often for the wrong reasons. But there is a significant controversy brewing up within the community of people using \u00a0set-theoretic methods (of which QCA is one example) as well. Recently, COMPASSS\u2026","rel":"","context":"In "Causality"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":336,"url":"http:\/\/re-design.dimiter.eu\/?p=336","url_meta":{"origin":915,"position":1},"title":"Explanation and the quest for 'significant' relationships. Part II","date":"February 22, 2012","format":false,"excerpt":"In Part I I argue that the search and discovery of statistically significant relationships does not amount to explanation and is often misplaced in the social sciences because the variables which are purported to have\u00a0effects\u00a0on the outcome cannot be manipulated. Just to make sure that my message is not misinterpreted\u2026","rel":"","context":"In "Causality"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1035,"url":"http:\/\/re-design.dimiter.eu\/?p=1035","url_meta":{"origin":915,"position":2},"title":"The problem with scope conditions","date":"September 12, 2019","format":false,"excerpt":"Posing arbitrary scope conditions to causal arguments leads to the same problem as subgroup analysis: the 'results' are too often just random noise.","rel":"","context":"In "Causality"","img":{"alt_text":"","src":"https:\/\/i1.wp.com\/re-design.dimiter.eu\/wp-content\/uploads\/2019\/09\/significant.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":206,"url":"http:\/\/re-design.dimiter.eu\/?p=206","url_meta":{"origin":915,"position":3},"title":"Is unit homogeneity a sufficient assumption for causal inference?","date":"December 6, 2011","format":false,"excerpt":"Is unit homogeneity a sufficient condition (assumption) for causal inference from observational data? Re-reading King, Keohane and Verba's bible on research design\u00a0[lovingly known to all exposed\u00a0as KKV] I\u00a0think\u00a0they regard unit homogeneity and conditional independence as alternative assumptions for causal inference. For example: \"we provide an overview here of what is\u2026","rel":"","context":"In "Causality"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":526,"url":"http:\/\/re-design.dimiter.eu\/?p=526","url_meta":{"origin":915,"position":4},"title":"Correlation does not imply causation. Then what does it imply?","date":"October 9, 2012","format":false,"excerpt":"'Correlation does not imply causation' is an adage students\u00a0from all social sciences are made to recite from a very\u00a0early age. What is less often systematically discussed is what\u00a0could be actually going on so that two\u00a0phenomena are correlated but not\u00a0causally related. Let's try to make a list: 1) The correlation might\u2026","rel":"","context":"In "Causality"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":57,"url":"http:\/\/re-design.dimiter.eu\/?p=57","url_meta":{"origin":915,"position":5},"title":"Inspiring scientific concepts","date":"October 16, 2011","format":false,"excerpt":"EDGE asks 159 selected intellectuals What scientific concept would improve everybody's cognitive toolkit? You are welcome to read the individual contributions which range from a paragraph to a short essay here. Many of the entries are truly inspiring but I see little synergy of bringing 159 of them together. Like\u2026","rel":"","context":"In "Causality"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/img.youtube.com\/vi\/2kotK9FNEYU\/0.jpg?resize=350%2C200","width":350,"height":200},"classes":[]}],"_links":{"self":[{"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/posts\/915"}],"collection":[{"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=915"}],"version-history":[{"count":3,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/posts\/915\/revisions"}],"predecessor-version":[{"id":918,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/posts\/915\/revisions\/918"}],"wp:attachment":[{"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=915"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=915"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=915"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}