Can a magic item be activated when someone/thing else is also holding it?

Can magic items that do not require attunement and only specify "While holding it you can…" to activate its effect (such as the Wand of Magic Missiles for example) be used when also held by someone else?

Consider a character doing the following during their turn:

  1. A character stands within reach of an ally holding the item.
  2. They use their object interaction to grab the item in a proper way (while the ally is still holding it).
  3. They activate the item with their action.
  4. They let go of the item (since dropping an item is considered "free"), which is still being held by the ally.

Would this also work if the ally was replaced with an enemy? Or if the item was in a holster or scabbard on either the activating character or someone else, or if it was just stuck in a rock?