Spells out off combat

How would you guys handle a player casting for example Charm Person on a high level wizard out of combat? It has V,S components and my thought is if the wizard, who would be knowledgeable about magic, would get a chance to interrupt the spell. Or would the start of casting the spell result in a combat timescale with inits rolled, making the wizard have a chance to ready?

This just happened last session and I didn’t let the wizard have a chance to interrupt, resulting in her spilling some pretty important secrets after a failed save. It felt a bit off as the level diff was big between the caster and the target.