But OECD tax expert Jeffrey Owens, speaking at a taxation conference jointly hosted by the Organisation for Economic Co-operation and Development (OECD) and the Confederation of Swedish Enterprise, predicted that Sweden would reintroduce the tax "within five years".
Anders Borg told listeners that while he understood the OECD's criticism "a tax has to be legitimate" in order to function properly.
"It is the one tax that Swedes consider most unfair, which is why we have removed it," he added.
The Organisation for Economic Co-operation and Development has criticized the Swedish government's decision to abolish residential property tax on the basis that Sweden needs to reduce tax on labour, not property.
"When we are sitting here in five years' time Sweden will have reintroduced property tax," said Jeffrey Owens.
While the pair failed to see eye to eye on the property tax issue, the OECD has praised the government for cutting taxes for low earners and for its reforms to the unemployment benefit system.
And Jeffrey Owens urged the government to continue making adjustments to the tax system.
"In general Sweden's tax system is well-designed and you are on the right track but above all you should not stop reforming your tax system. Tax reform is a work in progress in today's world," he said.