The loudest voices for reform are currently coming from the UK. Hundreds of thousands of people from eastern Europe have moved to Britain over the past decade, and Prime Minister David Cameron has openly said he wants to try to find ways to stop that happening. But is that doable? Roderick Parkes is a newly appointed research fellow at the Europe Research Programme at Sweden's Institute for International Affairs, he says that it can be done, but not in the way Cameron has said.
"The British Prime Minister has, perhaps foolishly, promised to limit the numbers of people coming in and that essentially leads you to a quota system," Parkes says. "You can't do that under EU law. What you can do is try to steer the numbers of people coming by fixing the welfare system, or whatever it might be. There are enormous possibilities to do that under UK and EU law, but that's not what the UK government is looking at."
For a long time European leaders furrowed their brows at the lack of internal migration within the European Union. If people could move across the USA in search of a job, why wouldn't they in Europe, they asked. But those days are over, says Peo Hansen, associate professor of political science at Linköping University.
"It has changed quite dramatically with the Eastern enlargement and the crisis", he told Radio Sweden, "and people then started to move, and that resulted in so may EU governments saying 'no, now is the time to scale back on free movement, to restrict it, maybe even scrap it'. Now the European Commission's frustration is with the member states. However, some countries might not be so negative to migration in general, but what they would like to scale back are the rights that come with free movement. They give the movers the right to certain social rights and even political rights. That's the key point. A lot of countries don't mind getting free labour, especially when it is highly skilled, but they are more careful when it comes to rights."
And those rights are one of things British Prime Minister David Cameron is looking at. Should EU citizens living and working in Britain have the same access to the National Health Service, for example. Sweden and the UK often see each other as partners in the EU, but the arguments here in Sweden are different, Peo Hansen says.
"I haven't heard any changes in the Swedish policy", he adds, "the irony was that former Swedish PM Göran Persson coined the phrase 'social tourism', but then Sweden was even more open to free movement than Britain and Ireland, because Sweden didn't have any restrictions whatsoever and it's been working quite ok. The debate in Sweden is mostly about the beggars, so-called. I think that is the big issue and that hasn't been dealt with at all from a higher level."
So, could we soon be seeing the end of the freedom of movement in the European Union? Roderick Parkes from Sweden's Institute for International Affairs says many other parts of the world are trying to copy the EU system.
"Frankly I think we have no choice but to stick to it", he says, "it's one of those things that actually does work. It's politically difficult, but it works."
But while Peo Hansen says it is difficult to look into the crystal ball and see if freedom of movement in the EU will still exist in five years time, change has already taken place: "The perception of it not least has changed dramatically. Just we now have this category 'EU-migrants' and that we have a large number of extreme far-right parties that drive that argument even further. When you start to talk about people in terms of migrants, it often equals that they are a problem. So in that sense, free movement has changed quite dramatically already."