Airplay works but remote app can’t find AppleTVs

I can’t figure out why my Apple TV remote app or built-in control center app for controlling the Apple TV won’t work. It cannot find an Apple TV.

I am on the same network, my iPhone is paired with the Apple TV it shows up in the device list and when I held the close to the Apple TV and offered a chance to pair it. I can also airplay videos and screen mirror to the AppleTV, so it is just something to do with the remote app. Any ideas how to get the remote app to work?