It could be something silly, like DNS on the edgerouterx still being set to the old ISP’s.
It could be that the ISP is still stabilising your line.
It could be that whenever the connection drops and reconnects upstream from the ERX, the ERX takes a few minutes to re-establish (or if your provider does not give you a static IP, recover from the external IP change).
Make sure you haven’t done something crazy like set the external port of the ERX to the same IP as the ISP equipment.
If you don’t have anything very complex, you could nuke the ERX and restart everything.
Or if your ISP runs on PPPOE, you could try bypassing the ISP router. (And also, if you’re using pppoe, make sure your MTU isn’t too high)