Mehmet Kaplan, of the Green Party, has said he did not know there were people present with views "diametrically opposed" to his own.
Prime Minister Stefan Löfven has met with Kaplan on Thursday, and writes to the media that it is "to be deeply regretted" that the housing minister was at the same event as extremists.
Turkish-born Kaplan was at a celebration organised by Swedish-Turkish organisation Tumsiad Sverige. But the Swedish leader of the "Grey Wolves" Turkish nationalist extremists was also present.
He claimed he only learned much later who was at the table, though newspaper Aftonbladet spoke with the Grey Wolves group in Sweden and they said Kaplan is well aware of their group and its leader. Kaplan denies the allegation.
"That's how he sees it but I don't know him and for me, it's very important to clearly distance myself from what they stand for," he told Swedish Radio.
Kaplan said he takes responsibility for not checking beforehand who was attending the dinner and said he promised the Prime Minister he would work to avoid similar situations in the future.
"We agreed that it is important that each minister must know exactly what type of event they are attending, and that they are careful when it comes to public events," he told Swedish Radio.
The culture minister, a fellow Green, and the Left leader have also raised their voices in criticism.
"I think it is extremely bad and extremely serious," Left party leader Jonas Sjöstedt told news agency TT.
Green Party Culture Minister Alice Bah Kunke says it makes her feel the party must consider its routines as to how it responds to invitations to events.