I’ve got such an strange behaviour in a Ubuntu 18.04.3 LTS with latest updates:
It’s a multihomed server with a public IP for public services and a private IP on a second adapter for comunications like SSH. I’ve set up an IPSec tunnel (pfSense) that is working fine, so that I can SSH to all my servers on that private LAN.
Periodically, after several days working fine it happens that:
1) I can’t connect to any service from my VPN IP to the private IP address
2) If I restart the VPN client or the VPN service it doesn’t solve
3) If I restart the SSH service it doesn’t solve. It also happens with nginx, for example. Service restarts don’t solve it.
4) I neither can connect to new services on new ports from the previously told IPs source/destination
5) When the problem is happening other servers in that private LAN are not affected and you can connect from those servers to the SSH service of this affected server without problems. For example: my monitoring software is not showing any alert because it can connect to the services on its private IP without any problem.
6) If I capture network traffic (tcpdump) I see that the first “SYN” packet is really reaching the server, but he server is not replying with the SYN/ACK packet as expected. So, the communication is never stablished.
7) I can’t find any error message anywhere
8) the static network route for VPN Network isn’t lost, ‘netstat -rn’ shows the same routes
9) the server is idle
10) It’s not using any local firewall like iptables
11) To solve it I just have to put the network interface down and up (ifconfig eth1 down; ifconfig eth1 up), or reboot the server.
- it can’t be a IPSec problem because of facts #5 and #6
- it can’t be related to the software of the service (SSH) because of fact #3
- it can’t be related to network routes because of facts #6 and #8
- it’s not a performance probem, because of fact #9
It looks like a TCP/IP network stack problem but it would happen to more people and it should be solved in patches.
Can any body help me, please? Thanks in advance!