Can I disable O365 members button from my home page?

I have a site for which I would like to manage access requests. I can control who can give permissions, but I cannot control who adds who to the O365 group (see image). This ties to our group mailbox/calendar. Is there a way to disable this button so that users cannot add random people to the group? Right now, my site members group is messy with individuals that do not belong because my employees add them via O365 group.

enter image description here