The man was finally made aware of the fact that he had his dog in tow when a startled witness managed to signal to him.
By the time the dog was attended to it was beyond saving and had to be put down.
The incident occurred on road 70 near Gräv in Dalarna in central Sweden.
The 84-year-old had taken a break at a rest stop in order to allow his pet Australian terrier to have a pee, according to the local Dalademokraten daily.
The man tied the small dog to the tow-bar at the back of his car while he sat in the driving seat to make a couple of phone calls, the newspaper reported.
Apparently forgetting his tied up canine friend, the elderly man then drove out of the rest stop and back on to the rural highway.
The man had managed to travel over three kilometres before the driver of the car behind succeeded in alerting him to his unfortunate oversight.
The dog was in a shocking state and there was little alternative but to put it out of its misery.
The 84-year-old is now suspected of cruelty to animals and gross negligence.