Captain’s Log, stardate 5718.3. The Enterprise is orbiting Elba Two, a planet with a poisonous atmosphere where the Federation maintains an asylum for the few remaining incorrigible criminally insane of the galaxy. We are bringing a revolutionary new medicine to them, a medicine with which the Federation hopes to eliminate mental illness for all time. I am transporting down with Mister Spock, and we’re delivering the medicine to Doctor Donald Cory, the governor of the colony.
So, at least this one TOS episode indicates that there is only one small facility which the Federation uses to house all the remaining criminally insane people in the galaxy. I think we can assume that by the galaxy, Kirk actually means the Federation. But as of that era, there apparently exists a medication that they believe will cure people of mental illness.
How much stock we want to put in one third season TOS episode I think can be debated – and crucially we never get any confirmation as to the long term success of the medication – but it is part of the canon.
There is also the Tantalus V penal colony from “Dagger of the Mind”. Before they beam down, Kirk tells McCoy that it’s more like a resort colony than a cage, though the doctor who ran the facility was using a machine to essentially brainwash both inmates and staff.
As for incarceration and rehabilitation in the 24th century, we know Tom Paris was at the New Zealand Penal Settlement when Janeway sprung him, with the approval of the Rehab Commission. When we see the settlement, the prisoners appear to be doing some sort of labour: one is carrying something, and Paris appears to be calibrating some sort of machinery. Granted, we don’t know exactly what he was doing or why. Maybe he was working on a project he volunteered for or even conceived himself, and was given access to the resources to carry it out.
Ro Laren was on the Jaros II penal colony after her court martial. She was sprung from that by Admiral Kennelly, and he claims it was difficult to do so.
Kasidy Yates was incarcerated for six months for aiding the Maquis, though there’s never any indication that the sentence isn’t purely punitive.
In “Blaze of Glory” we saw that after his capture in “For the Uniform”, Michael Eddington was being held aboard a station in a fairly small cell. He was still wearing civilian clothes. It’s possible he hadn’t yet been formally tried and convicted, though.