The title should have been something like “We all knew that AIs today can’t be used for legal advice, and we did a routine demonstration to show why”.
It’s not like the chatbot looked for ways the user breaks laws, which would be a news.
Someone who uses ChatGPT daily knows. This generation of chatbot won’t be able to work within all the details of NYC exceptions to go against laws elsewhere, which dominate the training data.