How to route players in the same “room/match” to the same server?

I have a system where clients are connected to one of many servers sitting behind a loadbalancer with WebSockets. No matter which backend server clients are connected to, I am using a messaging platform (something similar to Apache Kafka) to relay actions from one client to all other clients in other servers who are supposed to be in the same room/match.

Ideally, if all clients from the same room/match can be connected to the same host, I can get rid of this inter-host messaging layer overhead and simply handle data relays within the server, which can substantially lower latency and also reduce complexity.

My question is, how can I make clients be connected to the same host?

Do I have to forego any kind of reverse proxy or load balancer, but expose server IP/Port directly for each game? (I am using a load balancer/reverse proxy which randomly distributes clients to one of the available servers in the backend.) Or can I use a load balancer with special rules (e.g. HAProxy with special rules) that route players from the same room to the same backend server instance?

For instance, Fornite can have millions of active players at any given time, but only up to 100 players are needed to be grouped together for each match.