What happens if you Shapechange whilst swallowed?

I’m running a tier 4 game and ran into an interesting situation.

1 of my players had been swallowed by a gargantuan sized homebrew creature. On his turn he used the Shapechange spell to become a Dragon Turtle (also gargantuan) in size.

What do the rules say should happen in this situation? Does he burst out thereby killing the monster? Is he somehow shunted out? Does the spell fail? or does something else happen?