The Commission’s plan for reforming EU asylum policy is very ambitious. But can it work?

Note: A 3,000-word analysis of reform plans that are probably never gonna see the light of day anyways, based on simple arithmetics and not-so-simple simulations. Also, an excuse to do graphs. Re-posted from Eurosearch


The European Commission announced last Wednesday a new package of proposals designed to reform the EU asylum system. The proposals include compulsory redistribution of asylum applications among the EU member states. This is called ‘corrective allocation mechanism’ or ‘fairness mechanism’.

Countries would be allocated ‘reference shares’ of asylum applications, and the moment a country’s reference share is exceeded by 50%, an automatic system will set it that will send the excess asylum applicants to countries that have not attained their reference shares yet. If member states do not cooperate, they will have to pay a ‘solidarity contribution’ of €250,000 for every asylum application they refuse to process.

The proposal for compulsory redistribution backed by the threat of financial penalties (sorry, solidarity contributions) is very ambitious. But can it work? And I mean, can it work in a strictly technical sense, provided that the EU musters the political support to adopt the proposals[1], manages to make the member states comply with the rules, ensures that they don’t game the national asylum statistics, and so on – additional problems that are by no means trivial to solve. Let’s also leave aside for the moment the normative issue whether such a compulsory redistribution system is fair, to asylum seekers and to the EU member states. For now, the simpler question – can the system work even in the best of political and bureaucratic circumstances?

How is the system supposed to work?

This is how the system is supposed to work, or at least how I think it’s supposed to work, based on the available explanations (see here, here, here and the actual draft regulation here):

(1) Each country gets a ‘reference key’ based on the relative size of its population and the relative size of its GDP (the two factors weighted equally) compared to the EU totals. For example, in 2014 Germany had a population of 81,174,000 (16% of the EU population of 508,191,116) and a GDP of €2,915,650 million (21% of the EU GDP of €13,959,741 million[2]), which, when combined, make up for a reference key of 18.5%.

(2) This reference key is translated into reference shares (indicative shares of the total number of asylum applications[3] made in the EU that each member state is expected to receive) by multiplying the reference key with the total numbers of asylum applications registered in the EU in the preceding 12 months. For example, according to Eurostat[4], between 1 January 2014 and 31 December 2014 the total number of asylum applications received in the EU was 653,885, so Germany’s reference share for January 2015 (covering the period 1 February 2014 – 31 January 2015) would be 18.5% of 653,885 which equals 120,969 applications.

(3) If a member state receives a number of applications that exceeds by 50% its reference share, the excess applications are to be redistributed to member states that have not attained their reference shares yet. For example, Germany would have to receive more than 181,453 (150% of 120,969) applications during the current and preceding 11 months to trigger the relocation mechanism.

(4) The reference totals and references shares are updated constantly and automatically.

How would the system work if it had to be implemented at the end of 2015 already?

Let’s first do the simple arithmetics to see how the system would work if it had to enter into force in the last month of 2015. We can plug in the total number of asylum applications registered in the preceding 12 months (hence, between 1 December 2014 and 30 November 2015) to calculate the country’s reference shares. We can then see who did more and who did less than their fair (reference) share, and we can estimate the number of transfers that would be necessary to balance the system.

According to Eurostat, the total number of asylum applications received in the reference period was 1,281,560, so this number is used in the calculations that follow. The figure below shows the reference shares (in black), 150% of the reference shares (in grey) and the actual numbers of applications received during the entire 2015 (in red). We can see that Hungary, Sweden, Germany, Austria, Finland, Bulgaria, and Cyprus would have exceeded 150% of their reference shares. And between them they would have had 449,821 ‘excess’ applications for redistribution. All the other member states are potential recipients (with the exception of Belgium, Denmark, and Malta which have received numbers of applications exceeding their reference shares but with less than 50%). The total number of available ‘slots’ for transfer is 592,469. The UK has 145,750, France has 106,227, Spain has 91,589, Italy has 67,689, Poland has 54,456, and so on. Even Greece would have to accept 8,577 more applications to achieve its fair share of registered asylum applications[5].

asylum_application_and quotas12

To sum up, if the ‘fairness mechanism’ was to enter into force in December 2015, it would require the relocation of almost half a million asylum seekers across the continent, with some member states having to receive more than 100,000 additional applications to balance the system. More than one-third of all applications received in the EU during the preceding 12 months would have to be relocated.

If the UK[6], for example, would refuse to accept the additional asylum applications to fill up its reference share, it would be expected to pay ‘solidarity contributions’ to a maximum of €36,437,500,000 (more than 36 billion euros). (For comparison, the total gross UK contribution to the EU budget in 2015 was around €16 billion). The total pool of asylum applications to be relocated would be worth  €112,455,250,000 (that is, more than 112 billion euros)! For comparison, the total budget of the EU for 2015 was € 141.2 billion.

To my mind, the scale of the potential fines (sorry, ‘solidarity contributions’) is so big as to make their application totally unrealistic. Of course, it is the threat of fines that is supposed to make the member states cooperate, but to do their work, fines still have to realistic enough.

To sum up the argument so far, things don’t look very bright for the solidarity mechanism. But one might object that 2015 was exceptional, and that it is precisely this type of imbalances observed at the end of 2015 that the fairness mechanism is designed to avoid. Yet, unless the system starts with a clean slate[7], the existing imbalances accumulated over the past months would have to be corrected somehow. The analysis above shows the enormous scale of the corrections needed, if the system would have entered into force five months ago.

How would the system handle the 2015 flow of asylum applications?

We can also try to simulate how the fairness mechanism with compulsory reallocation would have handled the flow of asylum applications that the EU experienced during 2015 (provided that the member states cooperated fully). That is, we start with the situation as observed in January 2015, we calculate the reference shares and apply the necessary transfers to balance the system, and then we move forward to February 2015 observing the actual numbers of applications received in reality, balance the system again with the necessary transfers, and so on until the end of the year. (The script for the analysis and the simulation (in R) is available upon request.)

Running the simulation for the entire course of 2015 delivers good news and bad news (for the architects of the proposed mechanism). The good news is that the redistribution system does not get ‘choked up’ – that is, it does not run out of capacity to redistribute asylum applications received by some member states in excess of 150% of their reference shares to member states that have yet to reach their reference shares. The bad news is that in order to get and stay balanced, the system applying the ‘fairness mechanism’ needs to make approximately 500,000 transfers (that makes 37% of all asylum applications received in the EU during the year).

The figure below shows the number of transfers (in black) that need to be made per month, together with the actual number of asylum applications received in the EU as a whole during this period (in grey). Approximately 157,000 transfers must be made in the first month of the simulation (January 2015) to balance the system initially, and the remaining 343,000 are needed to keep it in balance for the rest of the year. The peak is in August, when more than 50,000 transfers must be made. (The bars are not stacked upon each other but overlap).


The next figure shows the distribution of transfers per member state. The blue bars that go below the horizontal line at zero indicate that the member state is a net ‘exporter’ of asylum applications, and the red ones that rise above the zero line indicate that the member state is a net receiver of transferred applications during the year, according to the simulation. (The parts of the bars colored in light blue and light red show the transfers made in the first month of the simulation.)

It is clear from the figure that Hungary, Sweden, Germany and Austria export the greatest number of applications, while Poland, Italy, Spain, France and the UK are expected to receive the most transfers. Some countries actually change their status in the course of the year from exporters to receivers of additional asylum applications (Denmark) and vice versa (Finland). Even Sweden and Hungary – countries that are big net exporters for most of the year have to receive additional applications during one or two months (see here the detailed plot of the experience of individual countries over the 12 months of the simulation).asylum_application_transfers2015

Despite the huge amount of transfers, not all member states handle a completely proportional burden of the total EU pool of asylum applications throughout the year. While the monthly transfers correct for gross imbalances and ensure that no country deals with more than 150% of its reference share, the system still leaves potential for significant differences across the member states. The figure below demonstrates this fact by showing the simulated number of asylum applications in red (actual applications received and simulated transfers) and the references shares (in black and grey). While the two sets of bars are much closer now, and the red one does not exceed the grey one for any country, member states still vary from fulfilling 70% of their reference shares (for example, Croatia, Portugal, Romania, and Slovakia) to fulfilling close to 150% of their reference shares (Germany, Belgium and Austria).[8]

asylum_application_sims_and quotas


To sum up, the proposed compulsory redistribution of asylum applications among the EU member states can reduce the current imbalances, but only at the price of a huge amount of transfers between the member states. With the proposed parameters, the mechanism would be able to handle even a great influx of asylum seekers as the one observed during 2015. However, under the 2015 scenario, half a million applications would have to be redistributed to make it work. An enormous amount of transfers between member states would be necessary to balance the system initially (unless the mechanism starts with a clean slate), and as many as 50,000 applications per month might have to be redistributed later (under a scenario similar to the one that actually occurred in 2015).

The reference period of 12 months used for calculating the countries’ reference shares must be updated and moved forward every month to ensure that the system retains enough capacity for redistribution. Otherwise, the ‘cushion’ provided by the fact that countries only export applications once they receive 50% more than their reference shares might not be enough to guarantee that there are enough ‘free slots’ in other member states. For the reference period to be updated fast, reliable and almost instantaneous information about the flows of asylum seekers to all member states must be available. Currently, the latest month for which Eurostat has data on the asylum applications received in all EU member states is December 2015: that is,  at the moment, the reference shares can be updated with at least a 4-month lag. This might be too slow to accommodate the rapidly changing flows of asylum seekers to Europe and might quickly grind the fairness mechanism to a halt.

The disadvantage of a relatively short reference period of 12 month that is constantly updated is that some member states might have to receive transferred applications at one point of time and then be eligible to redistribute applications to other countries just a few months afterwards. Such moving around of asylum seekers across the continent is of course highly undesirable, and costly as well.

Although the system might be able to correct the gross imbalances, it might still allow significant differences in the asylum application burden that different EU countries carry to persist. This fact requires attention to the way the ‘excess’ applications are to be distributed among the eligible member states that have not achieved their reference shares yet (since, typically there will be more available slots than requests for redistribution).

Finally, given the scale of required transfers to make the fairness mechanisms work, the size of the proposed penalties (solidarity contributions) for refusing additional applications is so huge as to be completely unrealistic. If under this mechanism member states are potentially liable for amounts that exceed their total annual contributions to the EU budget, there is little chance they will agree to participate in the mechanism in the first place.

All in all, while in principle the proposed fairness mechanism can work to reduce significantly the imbalances in the distribution of asylum applications across the EU, once the member states realize the amount of additional applications they might have to deal with under this policy, it is highly unlikely they will approve it. And certainly not with the current parameters regarding the reference periods, the references shares or the financial penalties.


[1] The Polish foreign minister already called the proposals an ‘April Fool’s Day joke.’

[2] The population and GDP estimates are based on statistics provided by Eurostat. GDP is in current prices and comes from the ‘tec00001’ database, in particular.

[3] In addition to asylum applications, the system will also take in to account the number of resettled persons. Eurostat however does not provide monthly data on resettlement. And the (annual) numbers of resettlements relative to asylum applications are so low (less than 1%) that we can ignore them in the analysis without much harm.

[4] The monthly statistics on asylum applications are available in the ‘migr_asyappctzm’ database. The version used in the analysis has been last updated on 6  May 2016

[5] Wait, what? Greece would have to receive more asylum applications? That’s right. Although hundreds of thousands (if not millions) of migrants have arrived on Greek territory in the past year and a half, in 2015 the Greek state has registered as asylum seekers only a negligible proportion of them. So, according to the official statistics (that would be used to run the fairness asylum distribution mechanism), Greece would have to register more applications and would be eligible to receive transfers from other member states until it reaches its fair share. I will leave it to you to judge whether this is a feature or a bug of the proposed system.

[6] The UK and Ireland, by the way, are invited but are not required to join the proposed system, even if the rest of the member states approve it.

[7] From the available documents, it does not seem to be the case that the fairness mechanisms will start with a clean state; that is, with a reference period not extending 12 months back.

[8] Curiously, Hungary appears to have made more transfers than actual applications received during the year according to the simulation, due to the huge fluctuations in the monthly amount of applications registered (which average 20,000 in the first 9 months of the year, but then drop to less than a thousand in the last three) and the moving reference period for calculating the reference shares.

Key numbers

Austria AT 2,0% 88.160 25.631 62.529 49.713 -51.796
Belgium BE 2,5% 44.665 32.039 12.626 -3.393 0
Bulgaria BG 0,9% 20.375 11.534 8.841 3.074 -4.676
Croatia HR 0,6% 205 7.689 -7.484 -11.329 5.510
Cyprus CY 0,1% 2.265 1.282 983 343 -850
Czechia CZ 1,6% 1.515 20.505 -18.990 -29.242 14.694
Denmark DK 1,5% 20.940 19.223 1.717 -7.895 1.159
Estonia EE 0,2% 230 2.563 -2.333 -3.615 1.838
Finland FI 1,3% 32.345 16.660 15.685 7.355 -9.319
France FR 14,2% 75.755 181.982 -106.227 -197.217 94.098
Germany DE 18,4% 476.510 235.807 240.703 122.799 -120.770
Great Britain (UK) GB 14,4% 38.795 184.545 -145.750 -238.022 127.292
Greece GR 1,7% 13.210 21.787 -8.577 -19.470 7.408
Hungary HU 1,3% 177.130 16.660 160.470 152.140 -184.043
Ireland IE 1,1% 3.270 14.097 -10.827 -17.876 9.674
Italy IT 11,8% 83.535 151.224 -67.689 -143.301 56.460
Latvia LV 0,3% 335 3.845 -3.510 -5.432 2.756
Lithuania LT 0,4% 320 5.126 -4.806 -7.369 3.674
Luxembourg LU 0,2% 2.505 2.563 -58 -1.340 431
Malta MT 0,1% 1.850 1.282 568 -72 -519
The Netherlands NL 4,0% 44.975 51.262 -6.287 -31.919 9.498
Poland PL 5,2% 12.185 66.641 -54.456 -87.777 47.751
Portugal PT 1,6% 900 20.505 -19.605 -29.857 14.694
Romania RO 2,5% 1.255 32.039 -30.784 -46.803 22.958
Slovakia SK 0,8% 325 10.252 -9.927 -15.054 7.345
Slovenia SI 0,3% 275 3.845 -3.570 -5.492 2.756
Spain ES 8,3% 14.780 106.369 -91.589 -144.774 76.221
Sweden SE 2,5% 162.455 32.039 130.416 114.397 -134.256


5 simple things to know about asylum policy in the European Union

Migration is quickly turning into the defining issue of our time. This might sound cliché, but is true. Not only does migration top the list of most important problems facing society, but it is also divisive in a way no other issue is. Unlike problems like inequality or the environment, immigration polarizes and divides opinions of ordinary people in a manner that cuts through social classes, education levels, age groups, and political affiliations. Divisions and bitter disagreements run even within families and close circles of friends. For no other issue do I see on my Facebook wall the full gamut of opinions ranging from strong rejection of migrants and refugees to their unconditional welcome and embrace. Most opinions of course fall somewhere in-between expressing, for example, support for `genuine’ refugees fleeing war but not for economic migrants, or for Christian but not for Muslim immigrants; yet, deep and important disagreements remain.

The current crisis with the influx of hundreds-of-thousands asylum-seekers in Europe in the summer of 2015 is only but the current episode of the unfolding migration drama. The crisis and the political responses to it bring powerful emotions in people: fear and compassion, anger and humility, empathy and contempt. Together with polarization, emotions further cloud the discussion of asylum policies and the right thing for the European countries to do in this situation. In response, I want to share five simple things I happen to know about asylum policies in the EU. I am by no means a specialist on the legal aspects of asylum or migration law. My expertise comes from two rather technical policy studies I have conducted on the aggregate patterns of asylum applications and country refugee recognition rates over the last decade, and on their relationships with the broader social and political contexts. (see here and here for the academic articles and here for a blogpost and visualization based on them)

1) The asylum policies of the countries members of the European Union still differ a lot. Despite a considerable body of EU legislation harmonizing national asylum policies, in effect these national policies have not converged to a common set of standards and rules. The differences concern the handling of asylum application procedures (e.g. their duration), the actual support provided by the state to the applicants during the procedure, the quality of the reception facilities, the rights and privileges gained after (and if) a refugee status is granted, the forms of alternative protection if a refugee status is not granted, and what happens to those who are refused any protection. Most importantly of all, however, the EU member states differ significantly with respect to their recognition rates (the share of applications that are granted the refugee status) even for applicants from the same country of origin. By implication, this means that different states apply rather different criteria when assessing the asylum applications.
These differences are crucial to understand why a joint common EU asylum application center does not seem politically feasible at the moment and is not even being discussed as an option to respond to the current crisis. Until such considerable differences exist, a truly single European policy on asylum would remain out of sight.

2) The strictness of national policies towards asylum-seekers matter relatively little for the asylum flows they receive. You can think that by tightening their asylum policies – making reception conditions worse, reducing support during the application and after, or lowering the recognition rate, countries can lessen the asylum application burden that they face. But in fact asylum flows tend to be relatively insensitive and unresponsive, at least in the short and medium terms, to the strictness of national asylum policies and to how low or high the national recognition rates are. So manipulating national policy is not an effective tool to divert (or attract) asylum application flows. The same goes for the effect of current economic conditions or the political climate in a country (for example, whether there is broad public and party support or opposition to migrants). Asylum flows are directed to a large degree by geographical convenience and existing transit networks, by hearsay and stereotypes to be affected by the details of national asylum policies or recognition rates. The implication of all that is that no single country can unilaterally isolate itself from the asylum flows coming to Europe. That being said, because asylum flows are highly clustered (see below), not being on what is at the moment the most convenient route to Western and Northern Europe can dramatically affect the number of asylum-seekers that pass through or end up in your country.

3) Asylum-seekers from the same nationality or region tend to cluster in particular places. Not only do asylum-seekers from the same region or country tend to travel on the same routes employing the same networks and middlemen, but they also tend to cluster when and where they choose a place to lodge an application and where they settle if allowed. These points are quite intuitive. Extended family ties and networks provide for crucial information about handling the asylum-application process and about the living and working conditions in the host country and city. They also provide support and protection, etc. So no wonder that new asylum-seekers and refugees try to go to where they family and friends already are.
What is important to recognize, however, are the not so obvious policy implications of the fact that asylum-seekers and refugees cluster in space. Because migrants would tend to congregate in few places, these places would be subject to a much greater asylum burden than others. This goes for countries, but also for cities and regions within countries.
That is why countries are reluctant to let asylum-seekers and refugees settle wherever they wish in the EU. Otherwise, the fear is that because of the attracting power of existing networks of relatives and compatriots, very few places will have to deal with the challenges of supporting and integrating a great proportion of the refugees. The call for mandatory country (and existing regional within-country) quotas are partly responding to these expectations.

4) Even when recognized as such, refugees do not enjoy a freedom of movement in the EU. As mentioned above, refugees (and asylum-seekers) are not allowed to move, reside and work freely within the EU, unlike citizens of its member states. Even though recognized refugees might have the rights to work and live in the country that has recognized their status and even benefit from the national social protection policies, they cannot choose to relocate to another member states. This is important in order to understand why it is so crucial for the asylum-seekers to reach the desired place in Europe before they lodge an application.
But it is also important to understand why the compulsory re-settlement based on country quotas that the European Commission proposes would likely not work. Even if adopted by the Council (which at the moment seems rather unlikely), the scheme would run into troubles the moment the refugees try to skip their imposed host countries and go to where their family and support networks are. And they will. The resettlement quote scheme would then have to be coupled with measures like compulsory self-reporting or tagging that would allow for tracing the location of refugees and asylum seekers. Such measures would not only by expensive, but morally objectionable as well.

5) Even when their requests for asylum are rejected, asylum seekers often stay in Europe. This is the dirty little secret of asylum policy in Europe. Even when an asylum application has been rejected, and even when other forms of alternative protection are not granted, the migrants are rarely sent back to their country of origin. They either disappear into illegality but never leave the continent or exist in a para-legal limbo where their presence is tolerated but no support is provided. European countries differ in the extent to which they allow this to happen, and it is hard to get precise numbers about the scale of the problem, but it is in any case huge. Alternatives are, however, hard to find as locating and sending people back to their country of origin is expensive, often impossible if the migrants lack proper documents, and, many would argue, morally objectionable. But this fact undermines the idea that asylum-seekers are a special group of migrants who are only allowed to stay in the country if they face serious threats for their lives and dignity at home. If those who are rejected are allowed or tolerated to stay anyways, the difference between an asylum-seeker and a migrant motivated by economic or other reasons is much hard to draw in the public mind. Note that I am not saying that people migrating for reasons others than fleeing wars and persecution should not be welcomed; only that many people have different attitudes and policy preferences with respect to different groups of migrants, and that blurring the boundaries between the groups can have negative consequences for people’s selective support of particular groups, like refugees.

 All in all, none of these five points suggest a comprehensive solution to the current asylum crisis or point to a clear way forward. What they do, hopefully, is to outline some of the facts and constraints that those in power must have in mind when designing responses to the situation and some arguments with which the judge existing proposals.

To put my cards on the table, I currently think that a combination of three policies can be preferable to the current system and to existing proposals:

  • A centralized single EU-managed system of asylum application centers at places along (and perhaps even outside) the borders of the EU, financed by the EU budged and staffed by European civil servants (support for the regions where the application centers are located would be needed to handle the flow of asylum-seekers during the time their applications are being assessed);
  • Free movement for recognized asylum seekers within the continent. This should include the rights to move, settle, and work, but not necessarily access to the social systems of the host countries. Places where refugees happen to cluster disproportionately would get support from a pot to which all countries contribute.
  • Strict control of the external borders of the EU to channel the applications through the official centers and strict `no access’ policy for people denied asylum or alternative forms of protection.

This would represent a rather drastic change from the system currently in place so it probably has low political feasibility. At the same time, current proposals do not seem to fare much better in the EU decision-making bodies, so the scale of required changes should be no reason to disregard the ideas.

Why political scientists should continue to (fail to) predict elections?

The results from the British elections last week already claimed the heads of three party leaders. But together with Labour, the Liberal Democrats and UKIP, there was another group that lost big time in the elections: pollsters and electoral prognosticators. Not only were polls and predictions way off the mark in terms of the actual vote shares and seats received by the different parties. Crucially, their major expectation of a hung parliament did not materialize as the Conservatives cruised into a small but comfortable majority of the seats. Even more remarkably, all polls and predictions were wrong, and they were all wrong pretty much in the same way. Not pretty.

This calls for reflection upon the exploding number of electoral forecasting models which sprung up during the build-up to the 2015 national elections in the UK. Many of these models were offered by political scientists and promoted by academic institutions (for example, here, here, and here). At some point, it became passé to be a major political science institution in the country and not have an electoral forecast. The field became so crowded that the elections were branded as ‘a nerd feast’ and the competition of predictions as ‘the battle of the nerds’. The feast is over and everyone lost. It is the time of the scavengers.

The massive failure of British polls and predictions has already led to a frenzy of often vicious attacks on the pollsters and prognosticators coming from politicians, journalists and pundits, in the UK and beyond. A formal inquiry has been launched. The unmistakable smell of schadenfreude is hanging in the air. Most disturbingly, some respected political scientists have voiced a hope that the failure puts a stop to the game of predicting voting results altogether and dismissed electoral predictions as unscientific.

mudde afonso

This is wrong. Political scientists should continue to build predictive models of elections. This work has scientific merit and it has public value. Moreover, political scientists have a mission to participate in the game of electoral forecasting. Their mission is to emphasize the large uncertainties surrounding all kinds of electoral predictions. They should not be in the game in order to win, but to correct on others’ too eager attempts to mislead the public with predictions offered with a false sense of precision and certainty.

The rising number of electoral forecasts done by political scientists has more than a little bit to do with a certain jealousy of Nate Silver – the American forecaster who gained international fame and recognition with his successful predictions of the US presidential elections. (By the way, this time round, Nate Silver got it just as wrong as the others). For once, there was something sexy about political science work, but the irony was, political scientists were not part of it. And if Nate, who is not a professional political scientist, can do it, so can we – academic experts with life-long experience in the study of voting and elections and hard-earned mastery of sophisticated statistical techniques. So the academia was drawn into this forecasting thing.

And that’s fine. Political scientists should be in the business of electoral forecasting because this business is important and because it is here to stay. News outlets have an insatiable appetite for election stories as voting day draws near, and the release of polls and forecasts provides a good excuse to indulge in punditry and sometimes even meaningful discussion. So predictions will continue to be offered and if political scientists move away somebody else will take their place. And the newcomers cannot be trusted to have the public interest at heart.

Election forecasts are important because they feed into the electoral campaign and into the strategic calculations of political parties and of individual voters. Voting is rarely an act of naïve expression of political preferences. Especially in an electoral system that is highly non-proportional, as the one in the UK, voters and parties have a strong incentive to behave strategically in view of the information that polls and forecasts provide. (By the way, ironically, the one prognosis that political scientists got relatively right – the exit poll – is the one that probably matters the least as it only serves to satisfy our impatience to wait a few more hours for the official electoral results.)

Hence, political scientists as servants of the public interest have a mission to offer impartial and professional electoral forecasts based on state of the art methodology and deep substantive knowledge. They must also discuss, correct and when appropriate trash the forecasts offered by others.

And they have one major point to make – all predictions have a much larger degree of uncertainty than what prognosticators want (us) to believe. It is a simple point that experience has been proven right times and again. But it is one that still needs to be pounded over and over as pollsters, forecasters and the media get easily carried away.

It is in this sense that commentators are right: predictions, if not properly bracketed by valid estimates of uncertainty, are unscientific and pure charlatanry.  And it is in this sense that most forecasts offered by political scientists at the latest British elections were a failure. They did not properly gauge the uncertainty of their estimates and as a result misled the public. That they didn’t predict the result is less damaging than the fact they pretended they could.

Since the bulk of the data doing the heavy-lifting in most electoral predictive models is poll data, the failure of prediction can be traced to a failure of polling. But pollsters cannot be blamed for the fact that prognosticators did not adjust the uncertainty estimates of their predictions. The tight sampling margins of error reported by pollsters might be appropriate to characterize the uncertainty of polling estimates (under certain assumptions) of public preferences at a point in time, but they are invariably too low when it comes to making predictions from these estimates. Predictions have other important sources of uncertainty in addition to sampling error and by not taking these into account prognosticators are fooling themselves and others. Another point forecasters should have known: combining different polls reduces sampling margins of error, but if all polls are biased (as they proved to be in the British case), the predictions could still be seriously off the mark.

Offering predictions with wide margins of uncertainty is not sexy. Correcting others for the illusory precision of their forecasts is tedious and risks being viewed as pedantic. But this is the role political scientists need to play in the game of electoral forecasting, and being tedious, pedantic and decidedly unsexy is the price they have to pay.

Immigration from Central and Eastern Europe fuels support for Eurosceptic parties in the UK

[Note: re-post from Eurosearch]

Combining political, demographic and economic data for the local level in the UK, we find that the presence of immigrants from Central and Eastern Europe (CEE) is related to higher voting shares cast for parties with Eurosceptic positions at the 2014 elections for the European Parliament. Evidence across Europe supports the connection between immigration from CEE and the electoral success of anti-Europe and anti-immigration political parties.

Immigration has become the top political issue in the UK. It played a pivotal role during the European Parliament elections in 2014 and it is the most-talked about issue in the build-up to the national elections in 2015.

The arrival of Eastern Europeans in the wake of the ‘Big Bang’ EU enlargement in 2004 and 2007  has a large part of the blame to take for the rising political salience of immigration for the British public. Figure 1 shows that ever since the EU accession of the first post-communist countries in 2004, immigration has been considered one of the two most important issues facing the country by a substantial proportion of British citizens, surpassing even concerns about the economy, except for the period between 2008 and 2012.

Data source: Standard Eurobarometer (59 to 82).
Data source: Standard Eurobarometer (59 to 82).

These popular concerns have swiftly made their way into the electoral arena. Some political parties like UKIP and BNP have taken strong positions in favor of restricting immigration and against the process of European integration in general. Others, like the Conservative party, have advocated restricting access of EU immigrants to the British labour market[5] while retaining an ambivalent position towards the EU. Parties with positions supportive of immigration and European integration have altogether tried to dodge the issues for fear of electoral punishment. Arguably, political and media attention to immigration (and East European immigrants in particular) have acted to reinforce the public concerns. In short, British voters care about and fear immigration, and political parties have played to, if not orchestrated, the tune.

But there is more to this story. In recent research we find evidence that higher actual levels of immigration from Central and Eastern Europe (CEE) at the local level in the UK are related to higher shares of the vote cast for Eurosceptic parties at the last European Parliament elections in 2014. In other words, British Eurosceptic parties have received, on average and other things being equal, more votes in localities with higher relative shares of East European residents.

The relationship is not easy to uncover. Looking directly into the correlation between relative local-level CEE immigration population shares and the local vote shares of Eurosceptic parties would be misleading. Immigrants do not settle randomly, but take the economic and social context of the locality into account. At the same time, this local economic and social context is related to the average support for particular parties. For example, local unemployment levels are strongly positively correlated with the  vote share for the Labour party, and the local share of highly educated people is strongly positively correlated with the vote share for the Greens (based on the 2014 EP election results). Therefore, we have to examine the possible link between CEE immigration shares and the vote for Eurosceptic parties net of the effect of the economic and social local contexts which, in technical terms, potentially confound the relationship.

In addition, immigrants themselves can vote at the EP elections and they are more likely to vote for EU-friendly parties. This would tend to attenuate any positive link between the votes of the remaining local residents and support for Eurosceptic parties. Lastly, the available local level immigration statistics track only immigrants who have been in the country longer than three months (as of 27 March 2011). Hence, they miss more recent arrivals, seasonal workers and immigrants who have not been reached by the Census at all. All these complications stack the deck against finding a positive relationship between the local presence of CEE immigrants and the vote for Eurosceptic parties. It is thus even more remarkable that we do observe one.

Figure 2: A scatterplot of the relative share of CEE immigrants from the local population versus the residual share of the vote cast for Eurosceptic parties (UKIP and BNP) at the 2014 EP elections

Figure 2 shows a scatterplot of the logged share of CEE immigrants from the local level population as of 2011 (on the horizontal axis) against the residual share of local level vote shares of Eurosceptic parties (UKIP and BNP) at the 2014 EP election (on the vertical axis). Each dot represents one locality (lower-tier council areas in England and unitary council areas in Wales and Scotland) and the size of the dot is proportional to the number of inhabitants. A few localities are labeled. The voting share is residual of all effects of the local unemployment level, and the relative shares of highly educated people, atheists, and non-Western immigrants in the population. In other words, the vertical axis shows the proportion of the vote for Eurosceptic parties unexplained by other social and economic variables.

The black straight line that best fits all observations is included as a guide to the eye. Its positive slope indicates that, on average, higher shares of CEE immigrants are related with higher Eurosceptic vote shares. Formal statistical tests show that the relationship is unlikely to be due to chance alone.

While the link is discernable from random fluctuations in the data, it is far from deterministic. Some of the localities with the highest relative shares of CEE immigrants, like Brent, have in fact only moderate Eurosceptic vote shares, and some localities with the highest share of the vote cast for Eurosceptic parties, like Hartlepool, have very low registered presence of CEE immigrants. Nevertheless, even if it only holds on average, the relationship remains substantially important.

Does this mean that people born in the UK are more likely to vote for Eurosceptic parties because they have had more contact with East Europeans? Not necessarily. Relationships at the level of individual citizens cannot be inferred from relationships at an aggregate level (otherwise, we would be committing what statisticians call ecological fallacy). In fact, there is plenty of research in psychology and sociology showing that direct and sustained contact with members of an out-group, like immigrants, can decrease prejudice and xenophobic attitudes. But research has also found that the sheer presence of an out-group, especially when direct contact is limited and the public discourse is hostile, can heighten fears and feelings of threat of the host population as well. Both mechanisms for the effect of immigration presence on integration attitudes – the positive one of direct contact and the negative one of outgroup presence – are compatible with the aggregate level relationship that we find. And they could well coexist – for a nice illustration see this article in the Guardian  together with the comments section.

Is it really the local presence of immigrants from Central and Eastern Europe in particular that leads to higher support for Eurosceptic parties? It is difficult to disentangle the effects of CEE immigrants and immigrants from other parts of the world, as their local level shares share are correlated. Yet, the relative share of non-Western immigrants from the local population appears to have a negative association with support for Eurosceptic parties across a range of statistical model specifications, while the effect of CEE immigrants remains positive no matter whether non-Western immigration has been controlled for or not.

There is also evidence for an interaction between the presence of immigrants from CEE and from other parts of the world. The red line in Figure 2 is fitted only to the localities that have lower than the median share of non-western immigrants. It is steeper than the black one which indicates that for these localities the positive effect of CEE immigrants on Eurosceptic votes is actually stronger. The blue line is fitted only to the localities with lower than the median share of non-western immigrants. It is sloping in the other direction which implies that in localities with relatively high shares of immigrants from other parts of the world, the arrival of East Europeans does not increase the vote for Eurosceptic parties.

It is interesting to note the recent statement by UKIP leader Nigel Farage that he prefers immigrants from form former British colonies like Australia and India to East Europeans. Focusing rhetorical attacks on immigrants from CEE in particular fits and makes sense in light of the story told above.

We (with Elitsa Kortenska) also find that CEE immigration increases Euroscepticism at the local level in other countries as well. In a recently published article (ungated pre-print here) we report this effect in the context of the referenda on the ill-fate European Constitution in Spain, France, and The Netherlands in 2005 and on the Treaty of Lisbon in Ireland in 2008. In ongoing work we argue that local level presence of CEE immigrants is systematically related to higher vote shares cast for Eurosceptic parties in Austria, The Netherlands, and France, in addition to the British case discussed in this post.

Why does this all matter? The process of European integration presupposes the right of people to move and work freely within the borders of the Union. This is not only a matter of convenience, but of economic necessity. People from regions experiencing economic hardship must be able to move to other EU regions with growing economies for economic integration to function. In an integrated economy like the EU or the US, a Romanian or a Greek must be free to seek employment in the UK or in Poland the same way an American living in Detroit is able to relocate to California in search of work and fortune.

This is especially true given the lack of large-scale redistribution between EU regions. Economic Integration creates regional inequalities. One way to respond is to redistribute the benefits of integration. Another is to allow people and workers to move where employment chances are currently high. If none of these mechanisms is available, economic and political integration are doomed. Therefore, if immigration within the EU indeed fuels Euroscepticism, as our study suggests, the entire European integration project is at risk.

Visualizing asylum statistics

Note: of potential interest to R users for the dynamic Google chart generated via googleVis in R and discussed towards the end of the post. Here you can go directly to the graph.

An emergency refugee center, opened in September 2013 in an abandoned school in Sofia, Bulgaria. Photo by Alessandro Penso, Italy, OnOff Picture. First prize at World Press Photo 2013 in the category General News (Single).

The tragic lives of asylum-seekers make for moving stories and powerful photos. When individual tragedies are aggregated into abstract statistics, the message gets harder to sell. Yet, statistics are arguably more relevant for policy and provide for a deeper understanding, if not as much empathy, than individual stories. In this post, I will offer a few graphs that present some of the major trends and patterns in the numbers of asylum applications and asylum recognition rates in Europe over the last twelve years. I focus on two issues: which European countries take the brunt of the asylum flows, and the link between the application share that each country gets and its asylum recognition rate.

Asylum applications and recognition rates
Before delving into the details, let’s look at the big picture first. Each year between 2001 and 2012, 370,000 people on average have applied for asylum protection in one of the member states of the European Union (plus Norway and Switzerland). As can be seen from Figure 1, the number fluctuates between 250,000 and 500,000 per year, and there is no clear trend. Altogether, during this 12-year period, approximately 4.5 million people have applied for asylum, which makes slightly less than one percent of the total EU population. Of course, this figure only tracks people who have actually made it to the asylum centers and filed an application – all potential refugees who have perished on the way, or have arrived but been denied the right of formal application, or have remained clandestine are not counted.


Figure 1 also shows the annual number of persons actually recognized as ‘refugees’ under the terms of the Geneva Convention by the European governments: a status which grants considerable rights and protection. This number is quite lower with an average of around 40.000 per year (in the EU+ as a whole) which makes for less than half-a-million in total for the 12 years between 2001 and 2012. While the overall recognition rate remains between 7% and 14%, there is considerable variation between the different European states both in the share from the asylum flows they receive, and in the national asylum recognition rates.

Who takes the brunt of the asylum burden?
Both the asylum flows and the recognition rates are in fact distributed highly unequally across the continent, and in a way that cannot be completely accounted for by the wealth of destination countries, former (colonial) ties between asylum sources and destinations, nor geographical distance. To compare the shares of the total European pool of asylum applications and recognitions that a destination country gets, I create the so-called ‘burden coefficient’. The ‘burden coefficient’ compares the actual share of asylum applications a country received in a year to its ‘fair’ share which is defined as its relative share of the annual  total EU+ GDP. Simply put, if a country accounts for 10% of the European GDP, it would have been expected to receive 10% of all asylum applications filed in Europe that year. Taking account of GDP adjusts the raw asylum application shares in view of the expectation that richer and more populous countries should bear a proportionally higher share of the total European asylum ‘burden’ than poorer and smaller states.


Figure 2 shows the (logged) burden coefficient for asylum application shares for each EU+ country, averaged over the period 2010-2012. The solid line at zero indicates an asylum applications share perfectly proportional to a  country’s GDP share (a ‘fair’ burden). Countries with positive values receive a higher share of all applications than implied by their GDP level, and countries with negative values receive a lower than their implied share. (The dotted lines show where a country that is doing twice as much / twice as little as expected would be). Clearly, Spain, Portugal, Italy and many (but not all) of the East European countries underdeliver while Cyprus, Malta, Greece, and several West European states (notably Sweden, Belgium, and Norway) take a disproportionately high  share of the total pool of asylum applications filed in Europe over the last few years. Note that these comparisons already take into account (correct for) the fact that most of the Southern and Eastern European countries are poorer (have lower GDP) than the ones in the Western and Northern parts of the continent.


The picture does not change much when we focus on actual asylum recognitions (under the terms of the Geneva Convention) instead of applications. Figure 3 shows the burden coefficient (again averaged over 2010-2012) for full status refugee recognitions in Europe. The country ranking is similar with a few important exception – Greece grants much fewer asylum recognitions than expected even after we account for the state of its economy; Austria and Switzerland join the ranks of states which do much more than their implied share; and, sadly, many more countries in fact underdeliver when it comes to full refugee status grants. (Note that some states offer alternative protection to those denied the full ‘Geneva Convention’ status but the forms and level of this protection differs significantly across the continent).

Are asylum application shares responsive to the recognition rate?
Given these rather significant discrepancies across Europe in how many asylum applications countries get, and how much protection they offer, it is natural to ask whether the applications shares and the recognition rates are in fact related. Do asylum seekers flock at the gates of the European states which are most generous in their recognition policy? Do low recognition rates deter potential refugees from applying in certain countries? Can the strictness of asylum policy be an effective policy tool shaping future application flows? A comprehensive statistical analysis shows that while application shares and recognition rates are associated, their responsiveness to each other is rather weak. Simply put, manipulating the recognition rates is unlikely to have big practical effects on the asylum application share a country receives, and changes in the applications rates only weakly affect state recognition rates. The details of the analysis are rather technical and can be found here, but a dynamic visualization can help illustrate the patterns.

The dynamic interactive chart linked here shows the relationship between asylum applications and asylum recognition rates for each EU+ country over the last 12 years (the chart cannot be embedded in this post due to WordPress policy, but there is a screenshot below). When you press ‘Play’ each dot traces the experience of one country over time. You can choose to observe all, select a single state to focus upon, or tick a couple to compare their experiences.


A movement of a dot (and the trace in leaves) in a horizontal direction means that the number of asylum applications received by a country increases while the recognition rates remains the same. Similarly, a vertical move implies a change in the recognition rate but a stable asylum application flow. A trajectory that follows a diagonal suggests a link between applications and recognition rates.

When paused, the state of the chart at each year shows the cross-sectional association between applications and recognition rates: it is easy to see that there is a (rather stable) weakly-strong positive relationship. But the trajectories of individual countries over time do not suggest that there is a temporal link between the two aspects of asylum policy for particular countries. For example, in the UK between 2001 and 2004 both the recognition rates and the applications fall, which would suggest strong responsiveness, but then the recognition rate moves up from 4% to almost 30% without any significant increase in applications. The trajectory of Denmark (try it out) exhibits something close to a dynamic link with rates depressing applications initially but then when they rise again, applications seem to pick up as well. Of course, asylum flows are driven by many other factors as well, so while suggestive, the patterns in the chart should be interpreted with care.


More comprehensive analyses of asylum policy in Europe addressing these questions and more are available in my published articles accessible here and here. The original data comes from the UNHCR annual reports. The dynamic chart is generated using Google Chart Tools through the googleVis library in R, you can find the code here. I found it useful to generate a simple version, adjust the settings manually, and then copy the final settings via the Google Chart’s Advanced Panel back to R.

Predicting movie ratings with IMDb data and R

It’s Oscars season again so why not explore how predictable (my) movie tastes are. This has literally been a million dollar problem and obviously I am not gonna solve it here, but it’s fun and slightly educational to do some number crunching, so why not. Below, I will proceed from a simple linear regression to a generalized additive model to an ordered logistic regression analysis. And I will illustrate the results with nice plots along the way. Of course, all done in R (you can get the script here).

The data for this little project comes from the IMDb website and, in particular, from my personal ratings of 442 titles recorded there. IMDb keeps the movies you have rated in a nice little table which includes information on the movie title, director, duration, year of release, genre, IMDb rating, and a few other less interesting variables. Conveniently, you can export the data directly as a csv file.

Outcome variable
The outcome variable that I want to predict is my personal movie rating. IMDb lets you score movies with one to ten stars. Half-points and other fractions are not allowed. It is a tricky variable to work with. It is obviously not a continuous one; at the same time ten ordered categories are a bit too many to treat as a regular categorical variable. Figure 1 plots the frequency distribution (black bars) and density (red area) of my ratings and the density of the IMDb scores (in blue) for the 442 observations in the data.


The mean of my ratings is a good 0.9 points lower than the IMDb scores, which are also less dispersed and have a higher peak (can you say ‘kurtosis’).

Data-generating process
Some reflection on how the data is generated can highlight its potential shortcomings. First, life is short and I try not to waste my time watching bad movies. Second, even if I get fooled to start watching a bad movie, usually I would not bother rating it on IMDb.There are occasional two- and three-star scores, but these are usually movies that were terrible and annoyed me for some reason or another (like, for example, getting a Cannes award or featuring Bill Murray). The data-generating process leads to a selection bias with two important implications. First, the effective range of variation of both the outcome and the main predictor variables is restricted, giving the models less information to work with. Second, because movies with a decent IMDb ratings which I disliked have a lower chance of being recorded in the dataset, the relationship we find in the sample will overestimate the real link between my ratings and the IMDb ones.

Take one: linear regression
Enough preliminaries, let’s get to business. An ordinary linear regression model is a common starting point for analysis and its results can serve as a baseline. Here are the estimates that lm provides for regressing my ratings on IMDb scores:

summary(lm(mine~imdb, data=d))

            Estimate Std. Error t value Pr(>|t|)    
(Intercept)  -0.6387     0.6669  -0.958    0.339    
imdb          0.9686     0.0884  10.957   ***
Residual standard error: 1.254 on 420 degrees of freedom
Multiple R-squared: 0.2223,	Adjusted R-squared: 0.2205

The intercept indicates that on average my ratings are more than half a point lower. The positive coefficient of IMDb score is positive and very close to one which implies that one point higher (lower) IMDb rating would predict, on average, one point higher (lower) personal rating. Figure 2 plots the relationship between the two variables (for an interactive version of the scatter plot, click here):


The solid black line is the regression fit, the blue one shows a non-parametric loess smoothing which suggests some non-linearity in the relationship that we will explore later.

Although the IMDb score coefficient is highly statistically significant that should not fool us that we have gained much predictive capacity. The model fit is rather poor. The root mean squared error is 1.25 which is large given the variation in the data. But the inadequate fit is most clearly visible if we plot the actual data versus the predictions. Figure 3 below does just that. The grey bars show the prediction plus/minus two predictive standard errors. If the predictions derived from the model were good, the dots (observations) would be very close to the diagonal (indicated by the dotted line). In this case, they are not. The model does a particularly bad job in predicting very low and very high ratings.


We can also see how little information IMDb scores contain about (my) personal scores by going back to the raw data. Figure 4 plots to density of my ratings for two sets of values of IMDb scores – from 6.5 to 7.5 (blue) and from 7.5- to 8.5 (red). The means for the two sets differ somewhat, but the overlap in the density is great.


In sum, knowing the IMDb rating provides some information but on its own doesn’t get us very far in predicting what my score would be.

Take two: adding predictors
Let’s add more variables to see if things improve. Some playing around shows that among the available candidates only the year of release of the movie and dummies for a few genres and directors (selected only from those with more than four movies in the data) give any leverage.

 summary(lm(mine~imdb+d$comedy +d$romance+d$mystery+d$"Stanley Kubrick"+d$"Lars Von Trier"+d$"Darren Aronofsky"+year.c, data=d))

                      Estimate Std. Error t value Pr(>|t|)    
(Intercept)           1.074930   0.651223   1.651  .  
imdb                  0.727829   0.087238   8.343  ***
d$comedy             -0.598040   0.133533  -4.479  ***
d$romance            -0.411929   0.141274  -2.916  ** 
d$mystery             0.315991   0.185906   1.700  .  
d$"Stanley Kubrick"   1.066991   0.450826   2.367  *  
d$"Lars Von Trier"    2.117281   0.582790   3.633  ***
d$"Darren Aronofsky"  1.357664   0.584179   2.324  *  
year.c                0.016578   0.003693   4.488  ***
Residual standard error: 1.156 on 413 degrees of freedom
Multiple R-squared: 0.3508,	Adjusted R-squared: 0.3382

The fit improves somewhat. The root mean squared error of this model is 1.14. Moreover, looking again at the actual versus predicted ratings, the fit is better, especially for highly rated movies – no surprise given that the director dummies pick these up.


The last variable in the regression above is the year of release of the movie. It is coded as the difference from 2014, so the positive coefficient implies that older movies get higher ratings. The statistically significant effect, however, has no straightforward predictive interpretation. The reason is again selection bias. I have only watched movies released before the 1990s that have withstood the test of time. So even though in the sample older films have higher scores, it is highly unlikely that if I pick a random film made in the 1970s I would like it more than a random film made after 2010. In any case, Figure 6 below plots the year of release versus the residuals from the regression of my ratings on IMDb scores (for the subset of films after 1960). We can see that the relationship is likely nonlinear (and that I really dislike comedies from the 1980s).


So far both regressions assumed that the relationship between the predictors and the outcome is linear. Needless to say, there is no compelling reason why this should be the case. Maybe our predictions will improve if we allow the relationships to take any form. This calls for a generalized additive model.

Take three: generalized additive model (GAM)
In R, we can use the mgcv library to fit a  GAM. It doesn’t make sense to hypothesize non-linear effects for binary variables, so we only smooth the effects of IMDb rating and year of release. But why stop there, perhaps the non-linear effects of IMDb rating and release year are not independent, why not allow them to interact!

summary(gam(mine ~ te(imdb,year.c)+d$"comedy " +d$"romance "+d$"mystery "+d$"Stanley Kubrick"+d$"Lars Von Trier"+d$"Darren Aronofsky", data = d)) 

PParametric coefficients:
                     Estimate Std. Error t value Pr(|t|)    
(Intercept)           6.80394    0.07541  90.225   ***
d$"comedy "          -0.60742    0.13254  -4.583   ***
d$"romance "         -0.43808    0.14133  -3.100   ** 
d$"mystery "          0.32299    0.18331   1.762   .  
d$"Stanley Kubrick"   0.83139    0.45208   1.839   .  
d$"Lars Von Trier"    2.00522    0.57873   3.465   ***
d$"Darren Aronofsky"  1.26903    0.57525   2.206   *  
Approximate significance of smooth terms:
                  edf Ref.df     F p-value    
te(imdb,year.c) 10.85  13.42 11.09

Well, the root mean squared error drops to 1.11 and the jointly smoothed (with a full tensor product smooth) variables are significant, but the added predictive value is minimal in this case. Nevertheless, the plot below shows the smoothed terms are more appropriate than the linear ones, and that there is a complex interaction between the two:


Take four: models for categorical data
So far we treated personal movie ratings as if they were a continuous variable, but they are not – taking into account that they are essentially an ordered categorical variable might help. But ten categories, while possible to model, would make the analysis rather unwieldy, so we recode the personal ratings into five categories without much loss of information: 5 and less, 6,7,8,9 and more.

We can first see a nonparametric conditional destiny plot of the newly created categorical variable as a function of IMDb scores:

The plot shows the observed density for each category of the outcome variable along the range of the predictor. For example, for a film with an IMDb rating of ‘6’, about 35% of the personal scores are ‘5’, a further 50% are ‘6’, and the remaining 15% are ‘7’. Remember that the plot is based on the observed conditional frequencies only (with some smoothing), not on the projections of a model. But the small ups and downs seem pretty idiosyncratic. We can also fit an ordered logistic regression model, which would be appropriated for the categorical outcome variable we have, and plot its predicted probabilities given the model.

First, here is the output of the model:

summary(polr(as.factor(mine.c) ~ imdb+year.c,  Hess=TRUE, data = d)
        Value Std. Error t value
imdb   1.4103   0.149921   9.407
year.c 0.0283   0.006023   4.699

    Value   Std. Error t value
5|6  9.0487  1.0795     8.3822
6|7 10.6143  1.1075     9.5840
7|8 12.1539  1.1435    10.6289
8|9 14.0234  1.1876    11.8079

Residual Deviance: 1148.665 
AIC: 1160.665

The coefficients of the two predictors are significant. The plot below shows the predicted probability of the outcome variable – personal movie rating – being in each of the five categories as a function of IMDb rating and illustrates the substantive scale of the effect.


Compared to the non-parametric conditional density plot above, these model-based predictions are much smoother and have ‘disciplined’ the effect of the predictor to follow a systematic pattern.

It is interesting to ponder which of the two would be more useful for out-of-sample predictions. Despite the fact that the non-parametric one is more faithful to the current data, I think I would go for the parametric model projections. After all, is it really plausible that a random film with an IMDb rating of 5 would have lower chance a getting a 5 from me than a film with an IMDb rating of 6, as the non-parametric conditional density plot suggests? I don’t think so. Interestingly, in this case the parametric model has actually corrected for some of the selection bias and made for more plausible out-of-sample predictions.

In sum, whatever the method, it is not very fruitful to try to predict how much a person (or at least, the particular person writing this) would like a movie based on the average rating the movie gets and covariates like the genre or the director. Non-linear regressions and other modeling tricks offer only marginal predictive improvements over a simple linear regression approach, but bring plenty of insight about the data itself.

What is the way ahead? Obviously, one would want to get more relevant predictors, but, unfortunately, IMDb seems to have a policy against web-scrapping from its database, so one would either have to ask for permission or look at a different website with a more liberal policy (like Rotten Tomatoes perhaps). For me, the purpose of this exercise has been mostly in its methodological educational value, so I think I will leave it at that. Finally, don’t forget to check out the interactive scatterplot of the data used here which shows a user’s entire movie rating history at a glance.

As you would have noted, the IMDb ratings come at a greater level of precision (like 7.3) than the one available for individual users (like 7). So a user who really thinks that a film is worth 7.5 has to pick 7 or 8, but its average IMDb score could well be 7.5. If the rating categories available to the user are indeed too coarse, this would show up in the relationship with the IMDb score: movies with an average score of 7.5 would be less predictable that movies with an average score of either 7 or 8. To test this conjecture, a rerun the linear regression models on two subsets of the data: one comprising the movies with an average IMDb rating between 5.9 and 6.1, 6.9 an 7.1, etc., and a  second one comprising those with an average IMDb rating between 5.4 and 5.6, 6.4 and 6.6, etc. The fit of the regression for the first group was better than for the second (RMSE of 1.07 vs. 1.11), but, frankly, I expected a more dramatic difference. So maybe ten categories are just enough.