Self-Preservation: How to DM NPCs that Love Living?

I have it in mind that humanoid NPCs do not “rage against the dying of the light” as much as they should. Combat encounters frequently result in a full-on massacre, when realistically a single fatality would instill an idea in everyone’s minds the gravity of the situation, resulting in either a terrified retreat or an unconditional surrender. Anyone who’s wounded should begin to favor the preservation of their own mortality over combat success. Obviously these rules do not apply when the NPCs find themselves forced against their wills.

Does anyone have experience running this style of campaign? How does a combat scene play out with these restrictions? How would experience be awarded? Can the game still be rewarding and fun?