Judges don’t tell a team their facts are wrong, the other team does. Judges decide which of the teams had better argumentation.
I find the judge symbolism interesting and compelling, but moderators are not judges. They're not making a judgement in place of the citizens by the end.
Rather than only letting two people debate, they could ask for clarifications. If you see them as press, and as representing the citizens, you may even think they have to to fulfill the press code and their responsibility.