That depends wether you’re applying a pulling force on it or just a sideways force and how big of an issue it’s going to be if and when it comes loose. Temperature swings affect metal and brick differently so the hole might start crumbling around the rod after some years and it might come loose. It’ll definitely never be as tight as it is now.
This might come a bit too late but why didn’t you just get threaded rod and use one of these instead?