What’s under the caulk? If it’s ceramic tile, you can probably get away with a brass cup brush in a drill. Test in an inconspicuous area first obviously, but it should be less likely to scuff up the surface as something like scotchbrite.
Brass brush on a drill would probably melt into plastic fast. But a handheld one would work ok, or steel if you need more persuasion and aren’t worried about scratches.