What Counts as a Damage Modifier?

I was posting a homebrew/houserule here for critique (found here: Two-Weapon Fighting Mechanic Alterations ) and a point of confusion that came up was what exactly 5e counted as a damage modifier. The person who answered stated that they usually don’t consider adding extra dice as a damage modifier and that got me curious. This left me wondering about what exactly can be affected with calling out specifically damage modifiers

I looked around for an answer to this, but couldn’t find anything that wasn’t about ability scores or other beginner type questions.

So my question is, what is normally considered a damage modifier? And what falls under the damage modifier umbrella term? Would calling out just damage modifiers affect both static and dice related damage (from spells, class abilities, weapons, abilities, etc) or would it have to be spelt out to affect dice added to the damage?