What happens if I let my conjuration wizard be able to target unwilling creatures with Benign Transposition?

The 6th level conjuration wizard feature Benign Transportation says:

Starting at 6th level, you can use your action to teleport up to 30 feet to an unoccupied space that you can see. Alternatively, you can choose a space within range that is occupied by a Small or Medium creature. If that creature is willing, you both teleport, swapping places.

Once you use this feature, you can’t use it again until you finish a long rest or you cast a conjuration spell of 1st level or higher.

My plan is to make this able to work on unwilling creatures as well. They get to do a Charisma saving throw and if they succeed, they are unaffected.

Would this be game-breaking or abusable?