Onesphore Rwabukombe, a 54-year-old ethnic Hutu, is accused of genocide, murder and incitement to commit both at the higher regional court in the western city of Frankfurt.
"Between April 11 and 15, 1994 the accused ordered and coordinated three massacres in which a total of at least 3,730 members of the Tutsi minority who had sought refuge in church buildings were killed," prosecutor Christian Ritscher read out from the charge sheet in court.
The trial is due to run until at least October. The accused could face life in prison if convicted.
The Rwandan government, and now German prosecutors, accuse him of involvement in the systematic killing campaign in 1994 which left an estimated 800,000 dead, most of them Tutsis.
Rwabukombe, a former mayor of the town of Muvumba in northeastern Rwanda who has lived in Germany since 2002, was arrested in July. The charges were filed with a court in Frankfurt on July 29. He listened impassively as the charges were presented.
Ritscher said Rwabukombe had acted "with the goal of racial extermination."
He described a massacre at a church in the village of Kabarondo, one of the three Rwabukombe is accused of ordering and coordinating.
"1,700 refugees were gathered there. The men were killed first with arrows and similar weapons," followed by women, children and the elderly "who were killed with grenades and firearms," Ritscher said.
"No one was to survive. Those who fled the church were slaughtered."
One of Rwabukombe's lawyers, Natalie von Wistinghausen, filed a motion disputing the charges related to Kabarondo citing procedural problems.
Beyond trials before Rwandan courts and the International Criminal Tribunal for Rwanda (ICTR) in Tanzania, several other European countries have tried Rwandan suspects over the genocide.
Germany has prosecuted suspected war criminals from the Nazi era and the former Yugoslavia for genocide but this is the first time it has tried someone for alleged links to the Rwandan mass murder.
Rwanda's Prosecutor General Martin Ngoga welcomed the trial.
"We are grateful to Germany," he told AFP.
German prosecutors say Rwabukombe allegedly called on three occasions in the first half of April 1994 for a pogrom against the Tutsis, with the result that a "countless" number of people were murdered.
He also stands accused of ordering a lower-ranking local official to turn away Tutsi refugees seeking shelter at his home, threatening otherwise to have his family killed. At least one of these refugees was subsequently murdered.
Around 50 witnesses have been called to testify at the trial.
According to non-governmental organisation Trial Watch, Rwabukombe was listed as number 435 on the most-wanted list of suspects responsible for the genocide in Rwanda before his arrest.
A top Rwandan genocide suspect, a former army captain nicknamed the "Butcher of Butare," went on trial Monday before the ICTR.
Captain Ildephonse Nizeyimana is accused of genocide, crimes against humanity and war crimes which he denied in earlier hearings after his arrest and transfer to the Tanzania-based court in 2009.