A quick literary search suggests there is no evidence linking a vegan diet with a reduction in testosterone. In fact, most studies found an small but significant increase in testosterone with specifically vegan men.
A simple google search shows nothing that backs up your claim, odd that you would have made it.