Are there pitfalls to implementing a hybrid site with both responsive layouts and dynamic serving layouts?

I have a 20 year old site that I want to upgrade to a responsive layout. Originally it was a desktop-only layout but when a significant portion of my user base went mobile I implemented dynamic serving layouts where I sniff the user agent and serve different HTML based on whether the user agent is for a mobile device or not.

I started to implement the responsive layouts, but I found that about 5% of my user base is on older browsers that don’t have all the CSS and JavaScript I would like to use. For example, only 95.42% of users fully support CSS grid layouts: While I wouldn’t want to take the time to develop a site just 5% of my users, I already have a site that works for those users and I don’t want to lose that much of my traffic when I move to responsive.

My current plan is to still do server side tests based on the user agent like this pseudo code:

use responsive-layout if (bot)  use desktop-layout if (msie or firefox < 54 or chrome < 58  or edge < 16) use mobile-layout if (opera-mini or android-browser or samsung-internet < 5 or safari-mobile < 10.3) use responsive-layout otherwise 

Most of my users and search engine crawlers would get the responsive layout. Only specific non-capable and older browsers would get my existing static layouts.

I know that Google supports either responsive layouts or dynamic serving layouts but I haven’t been able to find any information about a hybrid approach like this. Are there any pitfalls (especially with regards to SEO) of mostly using responsive but falling back to dynamic serving for some browsers?