"The discussions which have been going on for six months between Stieg Larsson's heirs and his former partner Eva Gabrielsson have ended," the writer's brother and father said in a statement on Tuesday.
"Unfortunately, she did not want to accept all or part of our proposal," they wrote.
The author of the three cult thrillers left behind a wrenching drama when he died suddenly of a heart attack at age 50 in 2004. He died without a will and since he and his partner of 32 years were not married and had no children, his estate went to his father and brother, in accordance with Swedish law.
That included royalties from the books -- and sale of film rights -- whose English titles are "The Girl with the Dragon Tattoo", "The Girl Who Played with Fire", and "The Girl Who Kicked the Hornet's Nest".
Millions of copies have been sold worldwide.
Even though Larsson and Gabrielsson were a couple for 32 years, she does not legally inherit as there is somewhat surprisingly no common law marriage in a generally progressive Sweden.
Swedish public opinion has been sympathetic towards the woman seen as his widow after being excluded from the riches of his publishing phenomenon.
According to Larsson's family, she has turned down their offers of a pay off.
In the autumn the pair offered Gabrielsson a 20 million kronor ($2.6 million) settlement. According to an often quoted estimate revenues from the books and films amount to more than 200 million kronor.
Rumours persist that Gabrielsson had an active role in the editing of the three novels chronicling the adventures of Mikael Blomkvist, a journalist like Larsson, and a young computer hacker and punky cohort in solving mysteries, Lisbeth Salander.
Gabrielsson has declined to comment on the rumours ahead of the publication
soon of a book she has written about her life with Larsson and reviewing what she argues to be the archaic legislation addressing common law partnership (sambolagen).