Who implements Cryptoki on PKCS#11 Infrastructure, Software or Token/Devices

I am new to PKCS#11, and I am going through the docs, and I am a little bit confused where Cryptoki fits in. And after I read the initial part of the OASIS Spec, I came to conclusion that the Implementation of the Interface is left to the application rather than the token – Smart Card, USB token or HSM. And, if that’s the case then how do you circumvent vender specific devices.