Are Imps incapable of speech or is my DM messing with me?

So, I am new to DnD and have been having a conversation with my DM about restrictions she is putting on my Warlock (pact of the chain), such as limiting my short rests to only one per day to restrict my spell slot regeneration. During this conversation she mentioned that she has given my familiar more agency than he should have and that per the rules he shouldn’t be able to talk.

According to the PHB, Imps have Common and Infernal as language skills. I don’t know where she found the rule that Imps can’t talk. Is this in the DM guide or is she making it up?