How far to go with semantic versioning?

Say you have an interface with a function List<A> doSomething(Collection<B> bs). Looking at Java’s List interface and its JavaDoc I could return an immutable List which doesn’t support addition, it could also be mutable but not allow null values (which some implementations actually don’t allow). Random access could be fast or slow (LinkedList).

I want to change the List implementation that is returned by doSomething which behaves differently from the currently returned List implementation but strictly speaking it’s a breaking change, isn’t it? I am really struggling with this as I don’t want to release a major version just for this little change.

But there is also Hyrum’s Law: [...] all observable behaviors of your system will be depended on by somebody. [...]

Where to draw the line here?