"I might take legal action. The moment he makes those comments, what can I do?" Pfister told Reuters by telephone.
Togo FA's secretary general Assogbavi Komlan said on Tuesday after the team's 2-1 defeat by South Korea in their opening Group G match that Pfister drank too much.
He also said a decision on who would coach the World Cup debutants through their next matches was due on Wednesday and that he did not want Pfister.
"I don't have time to deal with it now but after the World Cup I will also contact FIFA to see what we can do, just so we don't make any mistakes," Pfister said.
Pfister also repeated that he would remain as Togo coach until the end of the World Cup but did not know whether he would stay beyond that.
"I don't know yet. In this job you think from one day to the next, not beyond the World Cup," he said.
Pfister quit last week because he said he could not do his job with the squad embroiled in a pay dispute with Togolese officials.
Komlan's comments highlighted a split in the FA that is thought by media observers to be behind the messy situation over the coaching job in the past few days, with one faction supporting Pfister and the other wanting Mawuena in charge.
Togolese officials have said their players' demands for 155,000 euros (Rs 89,65,400) to play in the competition and 30,000 (Rs 17,35,500) each per win, or half that amount per draw, were too high.
Meanwhile, Togo defender Ludovic Assemoassa, who came off injured during their World Cup debut match, tore a tendon and had surgery in Frankfurt early on Wednesday, Pfister said.
He has been ruled out for the rest of the tournament.
Togo, who are at the bottom of Group G, face Switzerland in Dortmund on Monday and France four days later.