In Capture One, how to Color-tag one image at a time when multiple selected in viewer?

In Capture One, I can select multiple images at the same time and use the viewer to cycle through them.

I’d like to use this so that I can cycle through variations on the same image and pick the best one by either assigning green to the one I like or red to the ones I don’t like.

However, the - and + hotkeys for assigning color assigns the same color to all photos in the viewer, not just the currently selected. I can manually click to do it one at the time, but that is incredibly time consuming.

Is there a way to get the color tag hotkeys to work on one photo at a time?