Addressing a parliamentary committee, medical adviser Shigeru Omi said organisers should explain to the public why they are going ahead in the middle of a pandemic.
Games should go ahead to demonstrate to the world what can be achieved with the right plan and measures.
Towns and cities lined up to host Olympic training or events have increasingly expressed resistance
A top Japanese virologist and government adviser has warned of the risks of spreading COVID-19 infections during the Tokyo Olympics.
The report comes after two members of the Ugandan Olympic delegation tested positive for the coronavirus last month upon arrival in Japan. One tested positive at Narita airport near Tokyo, but the remainder of the team was allowed to travel to a training site.
Organisers have already banned overseas spectators and set a cap on domestic spectators at 50 per cent of capacity
Dozens of Japanese towns have abandoned plans to host Olympic athletes because of concern they will overburden stretched medical resources amid a fourth wave of coronavirus infections.
'If the Olympics were to spread infection, then what are the Olympics for?'
Organisers should be prepared to act swiftly to ban spectators or declare another state of emergency if needed, the experts said. If spectators are allowed, rules should be strict, such as limiting fans to local residents, the experts said.
'I believe we have already missed the opportunity to cancel ... We have been cornered into a situation where we cannot even stop now. We are damned if we do, and damned if we do not.'
The coronavirus has however infected increasing numbers of athletes and others involved with the Games, and authorities were on Friday trying to track down a Ugandan weightlifter who went missing from his training camp.