I found that to be the opposite. All of our clients have come to us via word of mouth and all of our initial clients were referrals from past colleagues. Because we jumped around so much we ended up with a very large network of ex colleagues that came to us for help once they knew we were available. As. those colleagues also switched jobs they knew to call us if they needed our help in their new place of employment.
They key is not to stay at one place for a long time but to make an impact during the time you spend there.