Explanation and the quest for ‘significant’ relationships. Part II

In Part I I argue that the search and discovery of statistically significant relationships does not amount to explanation and is often misplaced in the social sciences because the variables which are purported to have effects on the outcome cannot be manipulated.

Just to make sure that my message is not misinterpreted – I am not arguing for a fixation on maximizing R-squared and other measures of model fit in statistical work, instead of the current focus on the size and significance of individual coefficients. R-squared has been rightly criticized as a standard of how good a model is** (see for example here). But I am not aware of any other measure or standard that can convincingly compare the explanatory potential of different models in different contexts. Predictive success might be one way to go, but prediction is altogether something else than explanation.

I don’t expect much to change in the future with regard to the problem I outlined. In practice, all one could hope for is some clarity on the part of the researchers whether their objective is to explain (account for) or find significant effects. The standards for evaluating progress towards the former objective (model fit, predictive success, ‘coverage’ in the QCA sense) should be different than the standards for the latter (statistical & practical significance and the practical possibility to manipulate the exogenous variables).

Take the so-called garbage-can regressions, for example. These are models with tens of variables all of which are interpreted causally if they reach the magic 5% significance level. The futility of this approach is matched only by its popularity in political science and public administration research. If the research objective is to explore a causal relationship, one better focus on that variable and  include covariates only if it is suspected that they are correlated with the outcome and with the main independent variable of interest. Including everything else that happens to be within easy reach not only leads to inefficiency in the estimation. One should refrain from  interpreting causally the significance of these covariates altogether. On the other hand, if the objective is to comprehensively explain (account for) a certain phenomenon, then including as many variables as possible might be warranted but then the significance of individual variables is of little interest.

The goal of research is important when choosing the research design and the analytic approach. Different standards apply to explanation, the discovery of causal effects, and prediction.

**Just one small example from my current work – a model with one dependent and one exogenous time-series variables in levels with a lagged dependent variable included on the right-hand side of the equation produces an R-squared of 0.93. The same model in first differences has an R-squared of 0.03 while the regression coefficient of the exogenous variable remains significant in both models. So we can ‘explain’ 90% of the variation in the first case by reference to the past values of the outcome. Does this amount to an explanation in any meaningful sense? I guess that depends on the context. Does it provide any leverage to the researcher to manipulate the outcome? Not at all.

Slavery, ethnic diversity and economic development

What is the impact of the slave trades on economic progress in Africa? Are the modern African states which ‘exported’ a higher number of slaves more likely to be underdeveloped several centuries afterwards?

Harvard economist Nathan Nunn addresses these questions in his chapter for the “Natural experiments of history” collection. The edited volume is supposed to showcase a number of innovative methods for doing empirical research to a broader audience, and historians in particular. But what Nunn’s study actually illustrates is the difficulty of making causal inferences based on observational data. He claims that slave exports contributed to economic underdevelopment, partly through impeding ethnic consolidation. But his data is entirely consistent with a very different interpretation: ethnic diversity in a region led to a higher volume of slave exports and is contributing to economic underdevelopment today. If this interpretation is correct, it could render the correlation between slave exports and the lack of economic progress in different African states spurious – a possibility that is not addressed in the chapter.

The major argument of Nunn’s piece is summarized in the following scatterplot. Modern African states from which more slaves were captured and exported (correcting for the size of the country) between the XVth and the XIXth centuries are associated with lower incomes per capita in 2000 (see Figure 5.1 on p.162, the plot reproduced below is actually from an article in the Quarterly Journal of Economics which looks essentially the same):

The link grows only stronger after we take into account potential ‘omitted variables’ like geographical location, natural openness, climate, natural resources, history of colonial rule, religion and the legal system. Hence, the relationship seems more than a correlation and Nunn boldly endorses a causal interpretation: “the slave trades are partly responsible for Africa’s current underdevelopment” (p.165).

Not being a specialist in the history of slavery, my initial reaction was one of disbelief – the relationship seems almost too good to be true. Especially when we consider the rather noisy slave exports data which attributes imperfect estimates of slave exports to modern states which didn’t exist at the time when the slaves were captured and traded. While it is entirely plausible that slave exports and economic underdevelopment are related, such a strong association several centuries apart between the purported cause and its effect invites skepticism.

It seemed perfectly possible to me that the ethnic heterogeneity of a territory can account for both the volume of slave exports, and current economic underdevelopment. In my layman’s worldview, people are more likely to hunt and enslave people from another tribe or ethnicity than their own. At the same time, African countries in which different ethnicities coexist might face greater difficulties in providing public goods and establishing the political institutions conductive to economic prosperity. So I was a bit surprised that the analysis doesn’t control for ethnic diversity, in addition to size, climate, openness, etc.

But then towards the end of the essay, the relationship between slave exports and ethnic diversity is actually presented and the correlation at the country level turns out to be very high. But Nunn decides to interpret the relationship in the opposite direction: for him, slave exports caused ethnic diversity by impeding ethnic consolidation (which in turn contributes to economic underdevelopment today). He doesn’t even consider the possibility of reverse causality in this case, although the volume of slave exports could easily be a consequence rather than a cause of ethnic diversity in a region.

Of course, data alone cannot give an answer which interpretation is more likely to be correct. And this is exactly the point. When the assignment of countries into different levels of slave exports is not controlled by the researcher or randomized by nature, it is imperative that all possible interpretations consistent with the data are discussed and evaluated; especially in a volume which aims to bring research methodology lessons to the masses.

And finally, if my suggestion that ethnic diversity is more likely to be a cause rather than an effect of slave exports is correct, can ethnic diversity explain away the correlation between slave exports and economic performance? While Nunn doesn’t test this conjecture, he has the data available on his website, so why don’t we go ahead and check: while I can’t be entirely sure I replicate exactly what the original article is doing [there is no do-file online], a regression of income on slave exports with ethnic diversity included as a covariate takes the bulk of the significance of slave exports away.