Have default route from the faster interface, but run port forwarding through the slower one?

My computer is connected to the internet via two Ethernet interfaces – one is the faster one but doesn’t have a public IP assigned while the other does, but is slower. I’d like to be able to make connections to the machine via the slower interface, but when I’m making any new connections – use the faster one.

So far I tried the “ignore all advertised routes” option in DHCP settings for the slower interface and while Wireshark confirms that the slow interface gets the SYN packets, ACK gets sent over the faster one which has a different IP and the remote host (rightly so) resets the connection.

The router I’m using is running OpenWRT.

How can I solve my problem in Ubuntu 18.04 LTS, preferably via NetworkManager GUI?