Säpo has made the claim on the back of the large number of people who've escaped from the centre near Gothenburg in the past year as well as what they say is a lack of co-operation from the staff working there, one of whom has been found guilty of rioting during an EU protest in the city in 2001.
According to the Göteborgs-Posten newspaper, 50 people have escaped from the Kållered centre since 2008.
Lars Skoglund from the Western border police told the newspaper that the centre has been looked at for some time.
" We have long questioned the facility's operations and there have been several strange escapes."
The police and the staff at the centre were in open dispute in February this year when the police suspected that the deportation of a refugee, a Libyan man, was sabotaged by staff there.
Police later came in force with a warrant to take the man but were denied entry to the centre. The man was subsequently taken after the police threatened to break in.
The incident in February was reported but dropped due to lack of evidence.
One of the bosses of the detention centre Paul Fågelberg told TT news wires that he had total confidence in his staff.