I have a major in structural engineering, and I’m commenting to say that you want to be very careful with reasoning that something will total less force because it caps out at your body weight.
Just to give an example, if you have a wire strung between two poles, and you hang your 100 lb self from the middle of that wire, the tension against the poles can easily be thousands of pounds.
It’s not going to play out that way with the ladder, unless the ladder is flexing. But if you have a ladder that bends, and it’s got a 10 degree bend in it, while the ladder is straightening it can be exerting far more force than your weight on the two ends.
Considering the static force diagram, a 100 lb downward force is going to be balanced by a set of opposing forces that sum to an upward 100 lb force.
Given that there are horizontal forces involved too, these individual force components can easily be greater than their sum.
Not saying for sure this ladder situation is one of those situations, but if you’re consistently applying a heuristic that a system of forces will be limited to the force input of an external stressor, you can be very rudely surprised by the actual system when it generates forces in the tons.
I can’t declare this safe or unsafe, but really thick layers of “towel” between the upper end and the wall will help distribute the force on the wall making it less likely to punch through the drywall.
You could put a big piece of plywood there, but there’s no way to hold it in place.