Making aging relevant: How can I translate premature aging into a mechanical effect?

My group is about to come across a haunted sunken keep with an adversarial Ghost mini-boss. Being mostly composed of Humans (because who doesn’t want a free feat?), I expect them to be particularly vulnerable to Horrifying Visage’s aging effect.

The problem is, based on this question and my own reading of the PHB, it would seem to be that age is simply fluff. You can roleplay being any age you want and not have consequences mechanically.

The tone of the campaign is serious and leans toward deadly (though no one has died. Yet), and considering that you can only be prematurely aged by a Ghost by failing a save by 5 or more (a total roll of 8) the effect of aging should be a nuisance, at least.

I understand that 5th edition has done away with giving bonuses and penalties for old age as previous editions used to, trusting the GM to keep the tone of the game intact. For example, it would be up to the GM whether he would allow a Venerable Fighter to still function 100% normally, if that’s the kind of game you’re playing.

However, it doesn’t work for me. The venerable fighter has had years to perfect his swing in order to avoid that ache in his elbow, let alone avoid throwing his back. A magically aged character (or any aged character, if you’re running a realist game) who has prematurely aged as a result of effects such as Horrifying Visage has not had the years he needs to get used to the effects of aging and neither has he grown wiser or more intelligent.

How can I translate premature aging into a mechanical effect?

I am open to house-rules, but please back it up with experience of use in-game. Wild speculation and spit-balling are not good answers.