What happens when a creature that has swallowed another creature is polymorphed?

Twice this week, I have faced a situation where Polymorph was going to be cast on a creature that had swallowed a PC. As a DM, I ruled that if the polymorphed form was smaller, the swallowed PC would burst out, causing damage to the creature. As a player, my DM ruled that the spell would simply fail.

Are there any rules on what should happen in this scenario? The texts of Polymorph and the swallow ability of various monsters don’t seem to address this situation.

(Related, but for 4e, and Related, but in the other direction)