What are the consequences if wizards can cast unprepared spells from their spellbooks?

It’s a staple of the fantasy genre: faced with an obstacle the barbarian can’t punch his way through, the wizard flips through his spellbook until he finds the perfect spell. He reaches into his component pouch, withdrawing—somehow—exactly what he needs, then casts a powerful spell, surprising the heroes and allowing them to continue on.

Wizards don’t get to do that in 5e. They prepare so many spells per day out of their spellbook, and unless the other spells within are tagged ritual, they don’t get to see use until after the next long rest.

I want to house-rule that a wizard can cast unprepared spells from their book in the absence of exigent conditions. If I have time and space to crack out my spellbook, being disallowed from mage armor, disguise self, or jump without [8 minus sleep] hours of study feels arbitrary. What about the game changes, especially balance-wise, if wizards are allowed to cast unprepared spells from their spellbooks?

Note: this would be different from [ritual] spell casting from the spell book. In this proposed scheme, casting an unprepared spell would still require slots.