Is the Hunter Ranger’s extra damage from Colossus Slayer optional?

Can I choose to not use the Ranger’s Colossus Slayer feature when I hit an injured target with a bow and arrow?

When my DM found out about Colossus Slayer, he started making all my 1s on attack rolls a guaranteed hit against someone in my party as long as they’re in my range. We’re level 3 at the moment, and essentially half- and one-shotting my teammates every time I roll a 1 feels pretty sucky.