Only the workers outside the castle lost their memories. The guards had helmets made of a material that protected them from the worst effects, and the palace shielded those inside.
The end result is a caste system, where you have people who remember and give orders and people who forget and have no choice but to follow.