Can massive damage knock out rather than causing instant death?

Suppose, as happened last night, a character intends to subdue rather than kill an opponent. She made a melee attack, hit, declared her intent to knock out. The damage roll was enough to reduce the opponent to zero and the remaining damage exceeded the opponent’s maximum HP.

Now we would seem to have two rules invoked:

Instant Death. Massive damage can kill you instantly. When damage reduces you to 0 hit points and there is damage remaining, you die if the remaining damage equals or exceeds your hit point maximum. (PHB p.197)

Knocking a Creature Out. When an attacker reduces a creature to 0 hit points with a melee attack, the attacker can knock the creature out. The attacker can make this choice the instant the damage is dealt. The creature falls unconscious and is stable. (PHB p.198)

So which rule takes precedence? Or is it neither?