Portal frustum culling

Our Levels consists of rooms. Each room has an arbitrary number of portals into other rooms. We frustum cull rooms by testing whether all portal vertices (a portal consists of a single quad) are outside of the frustum this is done recursively until no room is found.

This however has one flaw. if a portal is visibly occluded, it is still taken into account because it is within the frustum. enter image description here

Is there an easy way to cull the rooms which are in the frustum but occluded? I had Hardware occlusion culling in mind, but I am not sure whether this is too overkill.