What are the security risks with rolling out your own Authorisation server implementation

Even though I’m working within .Net Core, this question is generally applicable to other platforms as well.

My question is to do with: Using a framework (such as IdentityServer) to manage implementation for Auth (Authentication/Authorisation) — Vs — rolling out your own implementation by following protocols. In this scenario, the ‘rolling out your own’ option wouldn’t rely on any middleware to manage auth – all the required endpoints/services/data-access would be self managed.

Specifically, I’d like to know: assuming you have followed the protocol specs, what are the security concerns when rolling out your own implementation?