{"id":331,"date":"2012-02-17T12:34:36","date_gmt":"2012-02-17T12:34:36","guid":{"rendered":"http:\/\/rulesofreason.wordpress.com\/?p=331"},"modified":"2012-02-17T12:34:36","modified_gmt":"2012-02-17T12:34:36","slug":"explanation-and-the-quest-for-significant-relationships-part-i","status":"publish","type":"post","link":"http:\/\/re-design.dimiter.eu\/?p=331","title":{"rendered":"Explanation and the quest for ‘significant’ relationships. Part I"},"content":{"rendered":"

The ultimate goal of social science is causal explanation*. The\u00a0actual goal of most\u00a0academic research is to discover significant relationships between variables. The two goals are supposed to be strongly related – by discovering (the) significant effects of exogenous (independent) variables, one\u00a0accounts for\u00a0the outcome of interest.\u00a0In fact, the working assumption of the empiricist paradigm of social science research is that the two goals are essentially the same – explanation is<\/strong> the sum of the significant effects that we have discovered. Just look at what all the academic articles with ‘explanation’, ‘determinants’, and ’causes’ in their titles do – they report significant effects, or associations, between variables.<\/p>\n

The problem is that\u00a0explanation and\u00a0collecting\u00a0significant associations\u00a0are not the same<\/strong>. Of course they are not. The point is obvious to all uninitiated into the quantitative empiricist tradition of doing research, but seems to be lost to many of its practitioners. We could have discovered a significant determinant of X, and still be miles (or even light-years) away from a convincing explanation of why and when X occurs. This is not<\/strong> because of the difficulties of causal identification – we could have satisfied all conditions for causal inference from observational data, but the problem still stays. And it would not go away\u00a0after we pay attention (as we should) to the\u00a0fact that statistical significance is not the same as practical significance.\u00a0Even the discovery of convincingly-identified causal effects, large enough to be of practical rather than only statistical significance, does not amount to explanation.\u00a0A successful\u00a0explanation needs to account for the variation in X, and\u00a0causal\u00a0associations need\u00a0not to – they might be significant but not even\u00a0make\u00a0a visible\u00a0dent\u00a0in the unexplained variation in X.\u00a0The difference I am talking about is partly akin to the difference between looking at the significance of individual regression coefficients and looking at the model fit as a whole (more on that will follow in Part II). The\u00a0current standards\u00a0of social science research tend to emphasize the former rather than the later which allows for significant relationships to be sold as explanations.<\/p>\n

The objection can be made that the discovery of causal effects is all we should aim for, and all we could hope for. Even if a causal relationship doesn’t account for large amounts of variation in the outcome of interest, it still makes a difference.\u00a0\u00a0<\/strong>After all, this is the approach taken in epidemiology, agricultural sciences and\u00a0other fields (like beer production) where the\u00a0statistical research paradigm has its origins. A pill might not treat all headaches but if it has a positive and statistically-significant effect, it will still help millions. But here is the trick – the quest for statistically significant relationships in epidemiology, agriculture, etc. is\u00a0valuable because all these effects can be considered as interventions – <\/em>the researchers have control over the formula of the pill, or the amount of pesticide, or the type of hops. In contrast,\u00a0social science researchers too often seek and discover significant relationships between\u00a0an\u00a0outcome and variables that couldn’t even remotely be considered as interventions. So we end up with a\u00a0pile of significant relationships which do not account for enough variation to count as a proper explanation and they have no value as interventions as their manipulation is beyond our reach. To sum up, observational social science has borrowed an approach to causality\u00a0which makes sense\u00a0for experimental research, and applied its standards (namely, statistical significance) to a context where the discovery of significant relationships is less valuable\u00a0because the\u00a0‘treatments’ cannot be manipulated. <\/strong>Meanwhile, what should really count – explaining when, how and why a phenomenon happens, is relegated to the background in the false belief that somehow the quest for significant relationships is a substitute.\u00a0It is like trying to\u00a0discover the fundamental function of the\u00a0lungs with epidemiological methods, and claiming success when you\u00a0prove\u00a0that cold air reduces significantly lung capacity. While the inference\u00a0might still be\u00a0valuable, it is no substitue for the original goal.<\/p>\n

In Part II, I will discuss what needs to be changed, and what can be changed in the current practice of empirical\u00a0social science research\u00a0to address the problem outlined above.<\/p>\n

*In my understanding, all explanation is causal. Hence, ‘causal explanation’ is tautology.\u00a0Hence, I am gonna drop the ‘causal’ part for the rest of the text.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

The ultimate goal of social science is causal explanation*. The\u00a0actual goal of most\u00a0academic research is to discover significant relationships between variables. The two goals are supposed to be strongly related – by discovering (the) significant effects of exogenous (independent) variables, one\u00a0accounts for\u00a0the outcome of interest.\u00a0In fact, the working assumption of the empiricist paradigm of social science research is that the two goals are essentially the same – explanation is the sum of the significant effects that we have discovered. Just look at what all the academic articles with ‘explanation’, ‘determinants’, and ’causes’ in their titles do – they report significant effects, or associations, between variables. The problem is that\u00a0explanation and\u00a0collecting\u00a0significant associations\u00a0are not the same. Of course they are not. The point is obvious to all uninitiated into the quantitative empiricist tradition of doing research, but seems to be lost to many of its practitioners. We could have discovered a significant determinant of X, and still be miles (or even light-years) away from a convincing explanation of why and when X occurs. This is not because of the difficulties of causal identification – we could have satisfied all conditions for causal inference from observational data, but the problem still stays. And it would not go away\u00a0after we pay attention (as we should) to the\u00a0fact that statistical significance is not the same as practical significance.\u00a0Even the discovery of convincingly-identified causal effects, large enough to be of practical rather than only statistical significance, does not amount to explanation.\u00a0A successful\u00a0explanation needs to account for…<\/p>\n

Continue readingExplanation and the quest for ‘significant’ relationships. Part I<\/span><\/a><\/div>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false},"categories":[8,33,1],"tags":[118,119,223,256,415,446,448,511,536,620,621],"jetpack_featured_media_url":"","jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p7g3hj-5l","jetpack-related-posts":[{"id":336,"url":"http:\/\/re-design.dimiter.eu\/?p=336","url_meta":{"origin":331,"position":0},"title":"Explanation and the quest for 'significant' relationships. Part II","date":"February 22, 2012","format":false,"excerpt":"In Part I I argue that the search and discovery of statistically significant relationships does not amount to explanation and is often misplaced in the social sciences because the variables which are purported to have\u00a0effects\u00a0on the outcome cannot be manipulated. Just to make sure that my message is not misinterpreted\u2026","rel":"","context":"In "Causality"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":526,"url":"http:\/\/re-design.dimiter.eu\/?p=526","url_meta":{"origin":331,"position":1},"title":"Correlation does not imply causation. Then what does it imply?","date":"October 9, 2012","format":false,"excerpt":"'Correlation does not imply causation' is an adage students\u00a0from all social sciences are made to recite from a very\u00a0early age. What is less often systematically discussed is what\u00a0could be actually going on so that two\u00a0phenomena are correlated but not\u00a0causally related. Let's try to make a list: 1) The correlation might\u2026","rel":"","context":"In "Causality"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":193,"url":"http:\/\/re-design.dimiter.eu\/?p=193","url_meta":{"origin":331,"position":2},"title":"Social science in the courtroom","date":"December 2, 2011","format":false,"excerpt":"Everyone who is interested in\u00a0the sociology of science, causal inferences from observational data,\u00a0employment gender discrimination, judicial sagas, or academic spats should read the latest issue of Sociological Methods & Research. The whole issue is devoted to the Wal-Mart Stores,Inc. v. Dukes et al. case - \"the largest class-action employment discrimination\u2026","rel":"","context":"In "Observational studies"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1035,"url":"http:\/\/re-design.dimiter.eu\/?p=1035","url_meta":{"origin":331,"position":3},"title":"The problem with scope conditions","date":"September 12, 2019","format":false,"excerpt":"Posing arbitrary scope conditions to causal arguments leads to the same problem as subgroup analysis: the 'results' are too often just random noise.","rel":"","context":"In "Causality"","img":{"alt_text":"","src":"https:\/\/i1.wp.com\/re-design.dimiter.eu\/wp-content\/uploads\/2019\/09\/significant.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":282,"url":"http:\/\/re-design.dimiter.eu\/?p=282","url_meta":{"origin":331,"position":4},"title":"Writing with the rear-view mirror","date":"February 2, 2012","format":false,"excerpt":"Social science research is supposed to work like this: 1) You want to explain a certain case or a class of phenomena; 2) You develop a theory and derive a set of hypotheses; 3) You test the hypotheses with data; 4) You conclude about the plausibility of\u00a0the theory; 5) You\u00a0write\u00a0a\u00a0paper\u2026","rel":"","context":"In "Academic publishing"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":206,"url":"http:\/\/re-design.dimiter.eu\/?p=206","url_meta":{"origin":331,"position":5},"title":"Is unit homogeneity a sufficient assumption for causal inference?","date":"December 6, 2011","format":false,"excerpt":"Is unit homogeneity a sufficient condition (assumption) for causal inference from observational data? Re-reading King, Keohane and Verba's bible on research design\u00a0[lovingly known to all exposed\u00a0as KKV] I\u00a0think\u00a0they regard unit homogeneity and conditional independence as alternative assumptions for causal inference. For example: \"we provide an overview here of what is\u2026","rel":"","context":"In "Causality"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/posts\/331"}],"collection":[{"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=331"}],"version-history":[{"count":0,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/posts\/331\/revisions"}],"wp:attachment":[{"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=331"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=331"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=331"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}