Suppose a creature is vulnerable to radiant damage, and is beginning its turn in the area of effect of Moonbeam. It will be taking radiant damage, and the damage roll is a 13. It succeeds on its saving throw, so the damage is halved, but it is vulnerable to radiant damage, so the damage is doubled.
Is the damage first halved, giving 6, and then doubled, giving 12? Or is it doubled first, then halved to give 13? Or do the effects cancel, for a total damage of 13?