How should I as a GM handle a player *character* who has a bad memory?

There is a player in my campaign who’s characters defining trait is that they have a bad memory.

I like creating mechanical representations of character traits if it makes sense to do so.

I’ve given the player a Homebrew feat called "bad memory" that basically says "roll disadvantage on any memory based intelligence checks".

My reasoning being that a character with bad memory should not have the same statistical probability of rolling well on a memory based check as a character with a good memory.

This player is adamantly fighting me on this though.

This is his reasoning:

The game already has the scaffolding and complexity in the locations where it is required. There are reasons why things like forgetfulness are included in the traits and features yet don’t have mechanics attached to them. If it was appropriate they would have been added by the devs. Mechanics which make it harder to use a character, and especially mechanics which remove abilities from players, are things which are only used in very rare cases and only ever for very short periods.

Note: This player is a very good role-player. It feels wrong to me to not represent the bad memory mechanically though.

How should I as a GM handle this situation?