Hideous Laughter while climbing

I found this question about hideous laughter being casted on someone who is flying. My question is for when it’s cast on someone when they‘re climbing.

The spell says fall prone, but it doesn’t say anything about them releasing their grip from what they’re holding (e.g. a ladder like what my PC in my campaign is climbing). Maybe I’m being pedantic but common sense would rule that, if they fall prone, they fall from whatever height they were at. No? Yes?