If you lean a ladder against a wall, only a small part of your weight pushes against the wall horizontally, most weight is still directed towards the ground. You can roughly estimate how much of your weight will push against the wall:
with a vertical ladder it would be 0.
with a ladder put up in a 45 degree angle it would be ~50% (actually less).
Your angle will be somewhere in between.
My guess is any wall can handle this, otherwise it would fall if you lean on it. But no one can tell you for sure without inspecting it.