Is starting an AWS instance with only ssh to port 22 insecure?

Unless someone has my private ssh key, how is leaving an aws instance open to but only on port 22 via ssh insecure?

enter image description here

The ssh key would be distributed to a small set of people. I prefer to not need to indicate their source IP addresses in advance.

I do see another similar question SSH brute force entry in aws ec2 instance .

If you disabled password based login via SSH, then it is very hard to brute force an SSH login using a private key (

Maybe this covers it? Just want to double check since in the security world you do not get a second chance.