How can giving skill checks critical fails and successes affect game balance?

At my table there is the house rule that skills can crit on a natural 1 and 20.

At least I think a natural 20 would be treated special over a 19. But I do know that 1’s give harsh consequences.

In my last session I rolled natural ones on 2 separate skill checks.

One was a Perception check to listen to the mutters of nearby NPCs, I rolled a natural 1(I had a +6 to it) and the DM said I was talking so loud that all the other PCs had to do their checks at disadvantage.

The second time I rolled insight to see if i could figure out about how badly our setting the dock ablaze affected an invading orc army, whose ship was also set on fire. I rolled a natural 1(+6) and the DM said I did not think it had any effect at all and was going to be frightened for the 1st 3 rounds of the next encounter.

I’m all for having fun and silly things with low skill rolls. And am ok with these harsh consequences if everyone at the table are for the most part. But I feel this kinda screws over bards and rogue who get expertise.

Continuation of another question.