{"id":282,"date":"2012-02-02T09:52:06","date_gmt":"2012-02-02T09:52:06","guid":{"rendered":"http:\/\/rulesofreason.wordpress.com\/?p=282"},"modified":"2012-02-02T09:52:06","modified_gmt":"2012-02-02T09:52:06","slug":"writing-with-the-rear-view-mirror","status":"publish","type":"post","link":"http:\/\/re-design.dimiter.eu\/?p=282","title":{"rendered":"Writing with the rear-view mirror"},"content":{"rendered":"

Social science research is supposed to work<\/strong> like this:
\n1) You want to explain a certain case or a class of phenomena;
\n2) You develop a theory and derive a set of hypotheses;
\n3) You test the hypotheses with data;
\n4) You conclude about the plausibility of\u00a0the theory;
\n5) You\u00a0write\u00a0a\u00a0paper with a structure (research question, theory, empirical analysis, conclusions) that\u00a0mirrors<\/strong> the steps above.<\/p>\n

But in practice, social science research often works<\/strong> like this:
\n1) You want to explain a certain case or a class of phenomena;
\n2) You test a number hypotheses with data;
\n3) You pick the hypotheses that matched the data best and combine them in a theory;
\n4) You conclude that this theory is plausible and relevant;
\n5) You write a paper with a structure (research question, theory, empirical analysis, conclusions) that\u00a0does not reflect<\/strong> the steps above.<\/p>\n

In short, an inductive\u00a0quest for a plausible explanation is masked and reported as deductive theory-testing. This fallacy is both well-known and\u00a0rather common\u00a0(at least in the fields of political science and public administration).\u00a0And, in my experience, it\u00a0turns out\u00a0to be tacitly supported by\u00a0the policies of some journals and reviewers.<\/p>\n

For\u00a0one of my previous research projects, I studied the relationship between public support and policy output in the EU. Since the state of the economy can influence both, I included levels of unemployment as a potential omitted variable in the empirical analysis. It turned out that lagged unemployment\u00a0is positively related to\u00a0the volume of policy output. In the paper, I mentioned this result\u00a0in passing but didn’t really\u00a0discuss it at length because 1) the original relationship between public\u00a0support and policy output was not affected, and 2) although highly statistically significant, the result was quite puzzling.<\/p>\n

When I submitted the paper at a leading political science journal, a large part\u00a0of the reviewers’ critiques focused on the fact that I\u00a0do not\u00a0have an explanation\u00a0for the link between unemployment and policy output in the paper. But why should I? I did not have a good explanation\u00a0why these variables\u00a0should be\u00a0related (with a precisely 4-year lag) when\u00a0I did the empirical analysis, so why pretend?<\/strong> Of course, I suspected unemployment as a confounding variable for the original relationship I wanted to study, so I took the pains of collecting the data and doing the tests,\u00a0still that certainly doesn’t count as an explanation<\/strong>\u00a0for the observed statistical\u00a0relationship between unemployment and policy output. But the point is, it would have been entirely possible to write the paper as if<\/strong> I had strong\u00a0ex ante\u00a0theoretical reasons to expect that\u00a0rising unemployment\u00a0increases the\u00a0policy output of the EU, and that the empirical test supports (or more precisely, does not reject<\/em>) this hypothesis. That\u00a0would certainly have\u00a0greased the\u00a0review process, and it only takes\u00a0moving\u00a0a few paragraphs from the concluding section to the theory part of the paper. So, if your data has a surprising story to tell, make sure it looks like you anticipated it all along – you even had a theory that predicted it! This is what I call ‘writing with the rear-view mirror’.<\/p>\n

Why is\u00a0it a problem? After all, an empirical association is an empirical association no matter whether you theorized about it beforehand or not. So where is the harm? As I see it, by pretending to have theoretically anticipated an empirical association,\u00a0you\u00a0grant it undue credence. Not only<\/strong> is data consistent with\u00a0a link between two variables, but\u00a0there are\u00a0strong theoretical<\/em> grounds to believe\u00a0the link\u00a0should be there.\u00a0A surprising statistical association, however robust, is just what it is – a surprising statistical association that possibly deserves speculation, exploration and further research. On the other hand, a robust statistical association ‘predicted’ by a previously-developed theory is way more – it is a claim that we understand how the world works.<\/p>\n

Until\u00a0journals and reviewers act as if proper science never deviates from the hypothetico-deductive\u00a0canon, writers will pretend that they follow it. While openly descriptive and exploratory research is frowned upon, sham theory-testing will prevail.<\/p>\n

Eventually,\u00a0my\u00a0paper on the links between public support, unemployment and policy output in the EU got accepted (in a different journal). Surprisingly given the bumpy review process, it has just been selected as the best article published in that journal during 2011. Needless to say, an explanation<\/strong> why unemployment might be related to EU policy output\u00a0is still wanting.<\/p>\n","protected":false},"excerpt":{"rendered":"

Social science research is supposed to work like this: 1) You want to explain a certain case or a class of phenomena; 2) You develop a theory and derive a set of hypotheses; 3) You test the hypotheses with data; 4) You conclude about the plausibility of\u00a0the theory; 5) You\u00a0write\u00a0a\u00a0paper with a structure (research question, theory, empirical analysis, conclusions) that\u00a0mirrors the steps above. But in practice, social science research often works like this: 1) You want to explain a certain case or a class of phenomena; 2) You test a number hypotheses with data; 3) You pick the hypotheses that matched the data best and combine them in a theory; 4) You conclude that this theory is plausible and relevant; 5) You write a paper with a structure (research question, theory, empirical analysis, conclusions) that\u00a0does not reflect the steps above. In short, an inductive\u00a0quest for a plausible explanation is masked and reported as deductive theory-testing. This fallacy is both well-known and\u00a0rather common\u00a0(at least in the fields of political science and public administration).\u00a0And, in my experience, it\u00a0turns out\u00a0to be tacitly supported by\u00a0the policies of some journals and reviewers. For\u00a0one of my previous research projects, I studied the relationship between public support and policy output in the EU. Since the state of the economy can influence both, I included levels of unemployment as a potential omitted variable in the empirical analysis. It turned out that lagged unemployment\u00a0is positively related to\u00a0the volume of policy output. In the paper, I mentioned this result\u00a0in passing…<\/p>\n

Continue readingWriting with the rear-view mirror<\/span><\/a><\/div>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false},"categories":[2,33,38],"tags":[55,249,256,320,489,497,530],"jetpack_featured_media_url":"","jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p7g3hj-4y","jetpack-related-posts":[{"id":123,"url":"http:\/\/re-design.dimiter.eu\/?p=123","url_meta":{"origin":282,"position":0},"title":"Academic fraud reaching new heights","date":"November 1, 2011","format":false,"excerpt":"Academic \u00a0fraud is reaching new heights lows. Dutch social psychologist Diederik Stapel (Tilburg University)\u00a0 is the culprit this time. A commission looking into the issue came up with a report [in Dutch] on Monday saying that \"the extent of fraud is very significant\" (p.5). Stapel fabricated data for at least\u2026","rel":"","context":"In "Academic publishing"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":577,"url":"http:\/\/re-design.dimiter.eu\/?p=577","url_meta":{"origin":282,"position":1},"title":"Hedging the bets: The US election outcome in the Dutch press","date":"November 7, 2012","format":false,"excerpt":"This is a guest post by Markus Haverland,\u00a0Professor\u00a0at\u00a0Erasmus University Rotterdam and author of a recent book on research methods. *** Causal knowledge about the world proceeds by testing hypotheses. The context of discovery precedes the context of justification. We all know that journalists and pundits often do it the other\u2026","rel":"","context":"In "Humour"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":455,"url":"http:\/\/re-design.dimiter.eu\/?p=455","url_meta":{"origin":282,"position":2},"title":"Review the reviews","date":"April 26, 2012","format":false,"excerpt":"Frank H\u00e4ge alerts me to a new website which gives you the chance to\u00a0review the reviews\u00a0of your journal submissions: On this site academic social science researchers have the opportunity to comment on the reviews they have received, and the process of decision-making about reviews, affecting articles submitted for publication, book\u2026","rel":"","context":"In "Academic publishing"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":193,"url":"http:\/\/re-design.dimiter.eu\/?p=193","url_meta":{"origin":282,"position":3},"title":"Social science in the courtroom","date":"December 2, 2011","format":false,"excerpt":"Everyone who is interested in\u00a0the sociology of science, causal inferences from observational data,\u00a0employment gender discrimination, judicial sagas, or academic spats should read the latest issue of Sociological Methods & Research. The whole issue is devoted to the Wal-Mart Stores,Inc. v. Dukes et al. case - \"the largest class-action employment discrimination\u2026","rel":"","context":"In "Observational studies"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":372,"url":"http:\/\/re-design.dimiter.eu\/?p=372","url_meta":{"origin":282,"position":4},"title":"Torture and game theory","date":"March 15, 2012","format":false,"excerpt":"The latest issue of Political Research Quarterly has an interesting and important exchange about the use of game theory to understand the effectiveness of torture\u00a0for eliciting truthful information. In this post I summarize the discussion, which is quite instructive for illustrating the prejudices and misunderstandings people have about the role\u2026","rel":"","context":"In "Game theory"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":11,"url":"http:\/\/re-design.dimiter.eu\/?p=11","url_meta":{"origin":282,"position":5},"title":"Rules of Reason, Reasons of Rules","date":"October 6, 2011","format":false,"excerpt":"This blog is about the uses of abuses of research on public policy and administration. It is about the Rules of Reason - the rules that guide the production of social science and that structure the design of academic research. But it is also about the Reasons of Rules -\u2026","rel":"","context":"In "Mission statement"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/posts\/282"}],"collection":[{"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=282"}],"version-history":[{"count":0,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=\/wp\/v2\/posts\/282\/revisions"}],"wp:attachment":[{"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=282"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=282"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/re-design.dimiter.eu\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=282"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}