After Germany lost 2-1 to the Czech Republic's 'B' side, the former World Cup striker moved towards the German supporters to applaud them.
The reaction was so hostile that Voeller did not stay long. He had songs written about him after guiding Germany to the 2002 World Cup final but is now in deep trouble.
"This is a bitter disappointment," said Voeller, looking close to tears at the post-match news conference.
"Two years ago, after the 2002 World Cup final, we celebrated in the locker room because we had something to be proud of.
"This is very sad, particularly because we had a great chance of going further in the tournament."
Euro 2004 has been a dismal tournament for the three times world and European champions who failed to survive the group stage of the tournament for the second time running.
Voeller has a contract until after the 2006 World Cup on home soil and said he wanted to stay on. He knows, however, there will be calls for him to stand aside.
"I'm realistic," he said. "You have to be when you exit in the first round. I have a contract until 2006 and assume I'll continue."
Since taking over from Erich Ribbeck after the Euro 2000 flop, Voeller restored the soccer-mad nation's pride with a surprise run to the 2002 World Cup final.
Problems soon returned as Germany struggled to qualify for Euro 2004 and suffered a poor run-up to the tournament including a 5-1 thrashing by Romania.
Apart from a convincing display in a 1-1 draw with the Netherlands in their opening match, Germany showed very little in Portugal and often appeared a one-man team, relying on Michael Ballack for inspiration and goals.
A dull 0-0 draw with Latvia put them in an uncomfortabe situation. But the players and coach had sounded confident they could prove themselves once again the ultimate tournament specialists in the decisive battle against the Czechs.
It was not to be. Germany looked short of ideas and failed to put their opponents under real pressure until the second half when substitute Milan Baros broke away to score the winner.
Germany have not beaten a European team at a major tournament since their 2-1 victory over the Czechs with a golden goal at the Euro 96 final.
They were again sterile in front of goal against the Czechs, apart from a piece of Michael Ballack magic in the first half, and paid a heavy price for their lack of firepower.
"We scored two goals in three games and that's not enough," said Voeller who was the classic poacher in his playing days.
"You need somebody up front to put them in."