custom layout applied from category not loading when accssing from direct URL

I am applying custom template updates inside admin pannel and selecting “apply to product” as “yes”. So far so good. When I load a product going all the way from navigation and selecting product from inside the category list, the layout update is applied accordingly.

The problem is that every time I load the product from its direct URL, the layout update does not load. I guess that magento is not finding the relation of the product to the category. (To reproduce this error you need to load from new private window)

My updates are all inside theme layout files, aka local.xml

Am I doing something wrong or is this a known bug?

Couldn’t find any info…

Thank you!