The Swedes dropped to third place in their qualifying group, two points behind Russia and only one ahead of Montenegro, while the Austrians advance to the 2016 European Championships.
Many were shocked by Sweden's collapse, which marked the team's biggest competitive home loss. Coach Erik Hamrén said it was a humiliating game.
"It hurts, all the way to the soul," he said according to newspaper Dagens Nyheter. "This was a match that we could win. And then, to lose it the way we will did."
Austria had pulled ahead 2-0 before the end of the first half with the first goal coming in the ninth minute. They would score twice more before Sweden's Zlatan Ibrahimovic finally put one back in extra time.
Sweden's next match will be against Liechtenstein on October 9.