We actually have less genetic variation than most animals. There was a lot of bottlenecking in the paleolithic. And what little we do have is still mostly confined to Africa, because the rest or the world shared common ancestry as we left our original continent.
Like, 1 in 200 people is colourblind, or something? I don't think that's a reasonable argument that we're not trichromats.