Thousands of incoming SSH connection attempts despite firewall lockdown

So my server is getting slammed with thousands of SSH login attempts. Fail2ban is catching and banning them – but my inbox it’s worrying me and it filling up inbox with alerts. Here’s a sample of what I’m seeing:

May 28 15:26:09 sshd[4908]: input_userauth_request: invalid user test [preauth] May 28 15:26:09 sshd[4908]: pam_unix(sshd:auth): check pass; user unknown May 28 15:26:09 sshd[4908]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=<<SNIPIP>> May 28 15:26:11 sshd[4908]: Failed password for invalid user test from <<SNIPIP>> port 41344 ssh2 May 28 15:26:11 sshd[4908]: Received disconnect from <<SNIPIP>> port 41344:11: Bye Bye [preauth] May 28 15:26:11 sshd[4908]: Disconnected from <<SNIPIP>> port 41344 [preauth] 

What’s interesting is when I attempt to connect using something like:
ssh test@myserverip -p 41344

my connection attempt eventually times out and I see no entry in auth.log – which is what I would expect to happen given I’ve locked the machine firewall down via UFW: (sidenote: I’m already running ssh on a non standard port)

ufw status verbose Status: active Logging: on (low) Default: deny (incoming), allow (outgoing), disabled (routed) New profiles: skip  To                         Action      From --                         ------      ---- Anywhere                   DENY IN     144.202.55.196 Anywhere                   DENY IN     188.53.140.190 Anywhere                   DENY IN     185.50.197.159 Anywhere                   DENY IN     206.189.197.133 Anywhere                   DENY IN     61.175.121.73 Anywhere                   DENY IN     8.30.124.149 Anywhere                   DENY IN     193.105.134.45 Anywhere                   DENY IN     139.129.14.230 Anywhere                   DENY IN     37.247.96.111 22                         DENY IN     Anywhere 2200/tcp                   ALLOW IN    Anywhere 25/tcp                     ALLOW IN    Anywhere 80,443/tcp (Nginx Full)    ALLOW IN    Anywhere 2246                       ALLOW IN    Anywhere 2812                       ALLOW IN    Anywhere 2247/tcp                   ALLOW IN    Anywhere 19999/tcp                  ALLOW IN    Anywhere 82/tcp                     ALLOW IN    Anywhere 22 (v6)                    DENY IN     Anywhere (v6) 2200/tcp (v6)              ALLOW IN    Anywhere (v6) 25/tcp (v6)                ALLOW IN    Anywhere (v6) 80,443/tcp (Nginx Full (v6)) ALLOW IN    Anywhere (v6) 2246 (v6)                  ALLOW IN    Anywhere (v6) 2812 (v6)                  ALLOW IN    Anywhere (v6) 2247/tcp (v6)              ALLOW IN    Anywhere (v6) 19999/tcp (v6)             ALLOW IN    Anywhere (v6) 82/tcp (v6)                ALLOW IN    Anywhere (v6) 

So my main question is, how is it possible that someone is even getting a chance to attempt a login on port 41344 when I can’t?