Azure Key Vault – hardware vs software protection

I was wondering if I correctly understand the difference between hardware and software protected keys.

Quoting the Applied Cryptography in .NET and Azure Key Vault (page 146 available on Google books)

Azure Key Vault Hardware Mode

When you configure Key Vault to work in hardware mode, you get the most benefit from the service because not only are keys stored in the hardware, but all operations such as encryption, decryption, and digital signatures are also performed on the device, which gives you the high level of protection when using Key Vault. The extra level of security that this affords does come at a cost as you need to use a premium service plan, but the additional cost gives you the extra protection that you would want in a production system.

Azure Key Vault Software Mode On the flip side, when you configure Key Vault to work in software mode, your keys are stored on the hardware, but any other operations, such as encryption, decryption, and digital signatures are performed outside of the HSM hardware using standard Azure compute virtual machines. Since there is less work on the HSM, you save money. From a software interface point of view, there is no difference in how you use Key Vault between hardware and software mode; the differences are transparent to a developer. When you are planning your testing and production environments for your software application, it is a good idea to use Key Vault in software mode for your testing environments as you can keep the costs low, and then use the hardware version for your production environment as this gives you the most significant level of protection.

In summary, my secret key is safe with hardware protection as long as the encryption key used to secure my secret key is not read from the HSM (which requires tampering with it and it leaves evidence). My secret key does not leave the HSM which performs all the operations using my secret key on its own. However, the software protection doesn’t have this extra security layer and my secret key is given away to Azure compute virtual machines, and my secret key could therefore be stolen without leaving any physical evidence whatsoever. Is that correct?