The downsides are having to report Capital Gain, and losing the ability to report Capital Loss.
You just weigh those costs (tax owing for capital gain, or loss of tax credit for capital loss) to see if they are sufficiently offset by the expected returns.