In a press release the working group singled out both governments for contributing to Assange's lengthy detention and said that Assange had the right to compensation.
News of the UN's Working Group on Arbitrary Detention ruling was leaked on Thursday and touted by Assange, his legal team as well as his supporters.
However, the panel's finding would likely have little bearing on Assange's legal troubles since the group's report is not legally binding and both the Swedish and British governments said the decision would not affect their investigations or police work.
"Mr. Assange has been subjected to different forms of deprivation of liberty: initial detention in Wandsworth prison which was followed by house arrest and his confinement at the Ecuadorian Embassy," the panel wrote.
"Having concluded that there was a continuous deprivation of liberty, the Working Group also found that the detention was arbitrary because he was held in isolation during the first stage of detention and because of the lack of diligence by the Swedish Prosecutor in its investigations, which resulted in the lengthy detention of Mr. Assange."
The panel went on to say that both the UK and Sweden should "assess the situation of Mr. Assange to ensure his safety and physical integrity, to facilitate the exercise of his right to freedom of movement in an expedient manner, and to ensure the full enjoyment of his rights guaranteed by the international norms on detention."
According to the AP news agency, one of the four voting members of the UN panel disagreed with its decision.
Assange said on Thursday that he should be allowed to leave the Ecuadorian embassy, where he has been holed up since 2012 to avoid extradition to Sweden, and called upon British and Swedish authorities to accept the UN panel's ruling.
Swedish prosecutors suspect Assange of rape, in connection with a visit he made to Sweden in 2010. Two seperate allegations of sexual harassment, in connection with the same visit, have been dropped. He has been wanted for questioning by Swedish authorities since, though he has always denied the allegations.
Assange has always claimed that if he traveled to Sweden to be questioned he would be indicted for espionage by the US for his work with WikiLeaks.
Reacting to the release of the panel's statement, a spokesman for the UK government told the AFP news agency that the UK "catagorically rejects" the panel's ruling, saying: "this changes nothing."
When reached for comment by Radio Sweden, both the Swedish Foreign Ministry and the Swedish Public Prosecutors office declined to be interviewed.