Are you sure you were not seeing only notifications from your phone mirrored onto your desktop and not actual downloads? Bear in mind that is one of KDE Connect's features, showing notifications on the desktop of stuff happening on the phone. It doesn't mean that what the notification is telling you is happening on the desktop.
Fairly certain, as I hadn't downloaded anything on my phone when away from my PC that would have produced a notification to mirror. Also, sorry, I should have mentioned from the start that I typically run my phone in Do Not Disturb and Silent to minimize any notifications whatsoever.
On further digging I did notice that it looks as though the files mentioned in the notifications may have been downloads, but from literally years ago in, I think, every case (e.g. from 2018!). I don't think I even had Connect installed back then to have caught those notifications. 🤨