For some it means leaving behind work internships and children changing schools.
Ibrahim Kurdi tells Swedish Radio's P4 Halmstad station that he felt settled in the town and does not want to leave.
"I don't want to move from Halmstad. The Arena Hotel is my home now and I want to take part in this society, learn the language, and visit the library."
"If they move me to some forest then I will not have these possibilities," he adds."They are treating us like monkeys and sending us to the forest."
Another asylum seeker, 14-year-old Iman, is also upset. He tells P4 Halland:
"People are unhappy because they don't know why they have to move, I have started a new life here and do not want to leave my friends."
Also in Hyltebruk in Halmstad, another 50 refugees have to move for the same reason.
The Migration Board has sent a letter stating that people have the chance to stay in Halmstad if they find themselves a place to live that they pay for themselves.
Only two people got on the bus that came to pick up 14 asylum seekers this afternoon. The Migration Board has said that the rest can stay over the weekend.