Bonus action attack ability score modifier (5E Question)

This may be confusing but currently in the campaign there is a warlock with a magic revolver that allows them to add eldritch invocations related to eldritch blast to the weapon. So agonizing blast adds damage to the weapon. If this weapon was to be used for a secondary attack that ‘You do not add your ability score modifier to the damage of these attacks, unless that modifier is negative.’ would this remove the bonus damage of ONLY the ability score modifier used to fire the gun, or would it remove both that and agonizing blast.