What are the consequences of removing the potential for critical hit extra damage on spell attacks?

Current table has a house rule that spells attacks can not deal extra damage through critical hits. They have the rule that you can critically succeed or fail on a natural 20 or 1 on saving throws.

I brought up how I was worried about how this would affect the balance of the game, fairly certain this would impair characters who are primarily casters in the long run. (Most of the PCs are half casters or not at all, save for mine and a couple others.)

I was told they are going with this ruling because it is more balanced, but I’m not convinced. Could someone with a more thorough understanding of the rules elaborate on how this could unbalance the game? Or am I just worried over nothing?

This is a pay to play table that I’ve already paid a subscription for so I’d like to not be told to just walk away as an answer. The DM has told me if there really is a balancing issue he’d see to fixing it.

2nd half of this question.