"I announce my resignation from the office of the federal presidency with immediate affect," Köhler said in Berlin.
He said the decision came after withering criticism of comments he made connecting Germany's military deployment in Afghanistan with the country's economy.
"This criticism had absolutely no justification," said the 67-year-old former head of the International Monetary Fund.
Looking emotional, Köhler asked for his supporters to understand his surprising resignation. The conservative Christian Democrats was nominated to be the country's largely ceremonial head of state in 2004 and re-elected in 2009.
"It was an honour for me to serve Germany," he said.
Chancellor Angela Merkel said she regretted his departure.
"I tried to get the president to change his mind but unfortunately I was unsuccessful," Merkel told reporters.
"I always worked very well together with Horst Köhler. He was an important adviser, particularly in the financial and economic crisis, with his large international experience. I will miss this advice."
Click here to see more photos of President Horst Köhler.
But his remarks just over a week ago prompted a furious debate about Germany’s military engagement – and whether Köhler had damaged the image of the NATO mission in Afghanistan.
Köhler began by saying that Germany was in the country, alongside its allies, to ensure its security and that it was good and proper for these issues to be openly and robustly discussed.
He then added: “But my estimation is that, on the whole, we are on the way to understanding, even broadly in society, that a country of our size, with this orientation toward foreign trade and therefore also dependence on foreign trade, has to be aware that when in doubt in case of an emergency, military deployment is also necessary to protect our interests.
For example, free trade routes, for example to prevent instability in a whole region, which certainly have an negative impact on our opportunities via trade, jobs and income. All of that ought to be discussed and I believe that we are not doing too badly.”
The remarks appeared to be a major departure from the political orthodoxy on the Afghanistan mission, which says the Bundeswehr is there to protect Germany from terrorist groups who would use the country as a base were it to descend into lawlessness or Islamist theocracy.
Köhler said his comments were "misunderstood" and that he was not referring to the mission in Afghanistan, where Germany has 4,500 troops in a NATO-led force fighting a Taliban-led insurgency.
But Wichard Woyke from the University of Münster told television station N-TV that he deserved the hefty criticism because he had spoken out of turn.
"It is not the president's job to get mixed up in political affairs but he did. He should not be surprised that the criticism was so severe," said Woyke.
According to the German constitution, the president of the upper house of parliament, the Bundesrat, will temporarily take over Köhler's duties. Currently that post is held by Bremen Mayor Jens Böhrnsen from the centre-left Social Democrats.
The German president is elected by the Federal Convention comprising members of both houses of parliament. His replacement must now be elected within the next 30 days.