Permissions revoke delay

I have a strange issue with permissions. We have a web application with custom claims provider (I do not know if this is connected with the issue). We have some site collections with many documents (all documents have broken permissions… I know it is not ok but it has to be that way). The issue is that when I assign a user to a group (this group exists ‘inside’ our custom claims provider) the user receives permissions immediately – he can open the document. But if I revoke permissions – user still can open the document for around 4 or 5 minutes. After that time the permissions are ‘truly’ revoked. What the…?