The family's claim was rejected but now the filmmakers have agreed to edit the film and to replace the original with a new version.
"Call Girl" is a fictionalized account of a scandal that rocked Sweden in the 1970s. Top government ministers were accused of buying sex from underage girls.
The film mixes fact with fiction, but the prime minister character bears a strong resemblance to the late Swedish Social Democrat leader Olof Palme.
For instance, the script incorporates quotes from Palme's speeches and television appearances.
In one scene, the prime minister has sex with an underage girl in a hotel room. That is the scene that upset Palme's widow and his three sons.
"We think it is completely unacceptable to spread such unfounded and insane rumours about my father supposedly taking advantage of underage girls or engaging in pedophilia," Olof Palme's son Mårten said in an interview with Swedish Television News last winter.
But the filmmakers insisted that Call Girl was inspired by real events and is not a documentary.
"I am an artist, a film director," said Call Girl director Mikael Marcimain. "I have created a drama."
However, the film uses many known historical facts and recreates 1970s Stockholm in painstaking detail, from the home décor to the paper candy-bags. All that makes it hard for the viewer to know where the facts end and the fiction begins.
In the new version, the controversial hotel scene has been edited following discussions between the film team and the Palme family's lawyers.
Now, the man in the hotel room cannot be identified and cannot be tied to Olof Palme.
While some argued that the Palmes' lawsuit infringed on artistic freedom, the family also received a lot of sympathy.
Criminologist Leif GW Persson, who worked at Sweden's National Police Board at the time that the scandal was being investigated, said that Palme never came close to the 1970s call-girl network. Rumours that he used prostitutes were unfounded, he said.
Call Girl is being shown at several international film festivals and is due to be released on DVD.
According to newspaper Dagens Nyheter the new, edited version will replace the original one