{"id":336,"date":"2012-02-22T20:11:45","date_gmt":"2012-02-22T20:11:45","guid":{"rendered":"http:\/\/rulesofreason.wordpress.com\/?p=336"},"modified":"2012-02-22T20:11:45","modified_gmt":"2012-02-22T20:11:45","slug":"explanation-and-the-quest-for-significant-relationships-part-ii","status":"publish","type":"post","link":"http:\/\/re-design.dimiter.eu\/?p=336","title":{"rendered":"Explanation and the quest for ‘significant’ relationships. Part II"},"content":{"rendered":"

In Part I<\/a> I argue that the search and discovery of statistically significant relationships does not amount to explanation and is often misplaced in the social sciences because the variables which are purported to have\u00a0effects\u00a0<\/em>on the outcome cannot be manipulated.<\/p>\n

Just to make sure that my message is not misinterpreted – I am not arguing for a fixation on maximizing R-squared and other measures of model fit in statistical work, instead of the current focus on the size and significance of individual coefficients. R-squared has been rightly criticized as a standard of how good a model is** (see for example here<\/a>). But I am not aware of any other measure or standard that can convincingly compare the explanatory potential of different models in different contexts. Predictive success might be one way to go, but prediction is altogether something else than explanation.<\/p>\n

I don’t expect much to change in the future with regard to the problem I outlined. In practice, all one could hope for is some clarity on the part of the researchers whether their objective is to explain (account for) or find significant effects. The standards for evaluating progress towards the former objective (model fit, predictive success, ‘coverage’ in the QCA sense) should be different than the standards for the latter (statistical & practical significance and<\/em> the practical possibility to manipulate the exogenous variables).<\/p>\n

Take the so-called garbage-can regressions, for example. These are models with tens of variables all of which are interpreted causally if they reach the magic 5% significance level. The futility of this approach is matched only by its popularity in political science and public administration research. If the research objective is to explore a causal relationship, one better focus on that variable and \u00a0include covariates only if it is suspected that they are correlated with the outcome and<\/em> with the main independent variable of interest. Including everything else that happens to be within easy reach not only leads to inefficiency in the estimation. One should refrain from \u00a0interpreting causally the significance of these covariates altogether. On the other hand, if the objective is to comprehensively explain (account for) a certain phenomenon, then including as many variables as possible might be warranted but then the significance of individual variables is of little interest.<\/p>\n

The goal of research is important when choosing the research design and the analytic approach. Different standards apply to explanation, the discovery of causal effects, and prediction.<\/p>\n

**Just one small example from my current work – a model with one dependent and one exogenous time-series variables in levels with a lagged dependent variable included on the right-hand side of the equation produces an R-squared of 0.93. The same model in first differences has an R-squared of 0.03 while the regression coefficient of the exogenous variable remains significant in both models. So we can ‘explain’ 90% of the variation in the first case by reference to the past values of the outcome. Does this amount to an explanation in any meaningful sense? I guess that depends on the context. Does it provide any leverage to the researcher to manipulate the outcome? Not at all.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

In Part I I argue that the search and discovery of statistically significant relationships does not amount to explanation and is often misplaced in the social sciences because the variables which are purported to have\u00a0effects\u00a0on the outcome cannot be manipulated. Just to make sure that my message is not misinterpreted – I am not arguing for a fixation on maximizing R-squared and other measures of model fit in statistical work, instead of the current focus on the size and significance of individual coefficients. R-squared has been rightly criticized as a standard of how good a model is** (see for example here). But I am not aware of any other measure or standard that can convincingly compare the explanatory potential of different models in different contexts. Predictive success might be one way to go, but prediction is altogether something else than explanation. I don’t expect much to change in the future with regard to the problem I outlined. In practice, all one could hope for is some clarity on the part of the researchers whether their objective is to explain (account for) or find significant effects. The standards for evaluating progress towards the former objective (model fit, predictive success, ‘coverage’ in the QCA sense) should be different than the standards for the latter (statistical & practical significance and the practical possibility to manipulate the exogenous variables). Take the so-called garbage-can regressions, for example. These are models with tens of variables all of which are interpreted causally if they reach the magic…<\/p>\n

Continue readingExplanation and the quest for ‘significant’ relationships. Part II<\/span><\/a><\/div>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false},"categories":[8,33],"tags":[119,121,223,256,415,446,447,448,511,536,620,621,666],"jetpack_featured_media_url":"","jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p7g3hj-5q","jetpack-related-posts":[{"id":331,"url":"http:\/\/re-design.dimiter.eu\/?p=331","url_meta":{"origin":336,"position":0},"title":"Explanation and the quest for 'significant' relationships. Part I","date":"February 17, 2012","format":false,"excerpt":"The ultimate goal of social science is causal explanation*. The\u00a0actual goal of most\u00a0academic research is to discover significant relationships between variables. The two goals are supposed to be strongly related - by discovering (the) significant effects of exogenous (independent) variables, one\u00a0accounts for\u00a0the outcome of interest.\u00a0In fact, the working assumption of\u2026","rel":"","context":"In "Causality"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":526,"url":"http:\/\/re-design.dimiter.eu\/?p=526","url_meta":{"origin":336,"position":1},"title":"Correlation does not imply causation. Then what does it imply?","date":"October 9, 2012","format":false,"excerpt":"'Correlation does not imply causation' is an adage students\u00a0from all social sciences are made to recite from a very\u00a0early age. What is less often systematically discussed is what\u00a0could be actually going on so that two\u00a0phenomena are correlated but not\u00a0causally related. Let's try to make a list: 1) The correlation might\u2026","rel":"","context":"In "Causality"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":206,"url":"http:\/\/re-design.dimiter.eu\/?p=206","url_meta":{"origin":336,"position":2},"title":"Is unit homogeneity a sufficient assumption for causal inference?","date":"December 6, 2011","format":false,"excerpt":"Is unit homogeneity a sufficient condition (assumption) for causal inference from observational data? Re-reading King, Keohane and Verba's bible on research design\u00a0[lovingly known to all exposed\u00a0as KKV] I\u00a0think\u00a0they regard unit homogeneity and conditional independence as alternative assumptions for causal inference. For example: \"we provide an overview here of what is\u2026","rel":"","context":"In "Causality"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":220,"url":"http:\/\/re-design.dimiter.eu\/?p=220","url_meta":{"origin":336,"position":3},"title":"Slavery, ethnic diversity and economic development","date":"December 14, 2011","format":false,"excerpt":"What is the impact of the slave trades on economic progress in Africa? Are the modern African states which 'exported' a higher number of slaves more likely to be underdeveloped several centuries afterwards? Harvard economist Nathan Nunn addresses these questions in his chapter for the \"Natural experiments of history\" collection.\u2026","rel":"","context":"In "Development"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/re-design.dimiter.eu\/wp-content\/uploads\/2011\/12\/slave-trades.jpg?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":57,"url":"http:\/\/re-design.dimiter.eu\/?p=57","url_meta":{"origin":336,"position":4},"title":"Inspiring scientific concepts","date":"October 16, 2011","format":false,"excerpt":"EDGE asks 159 selected intellectuals What scientific concept would improve everybody's cognitive toolkit? You are welcome to read the individual contributions which range from a paragraph to a short essay here. Many of the entries are truly inspiring but I see little synergy of bringing 159 of them together. Like\u2026","rel":"","context":"In "Causality"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/img.youtube.com\/vi\/2kotK9FNEYU\/0.jpg?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":193,"url":"http:\/\/re-design.dimiter.eu\/?p=193","url_meta":{"origin":336,"position":5},"title":"Social science in the courtroom","date":"December 2, 2011","format":false,"excerpt":"Everyone who is interested in\u00a0the sociology of science, causal inferences from observational data,\u00a0employment gender discrimination, judicial sagas, or academic spats should read the latest issue of Sociological Methods & Research. The whole issue is devoted to the Wal-Mart Stores,Inc. v. Dukes et al. case - \"the largest class-action employment discrimination\u2026","rel":"","context":"In "Observational studies"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/posts\/336"}],"collection":[{"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=336"}],"version-history":[{"count":0,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/posts\/336\/revisions"}],"wp:attachment":[{"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=336"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=336"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=336"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}