What does it mean to delete a file that’s “shared with me” in Google Drive?

I don’t use Google Drive very much, but I noticed I had a couple of files in the “shared with me” section of the Google Drive web app. So I decided to remove them from view, and the only way to do that is to delete them, one by one.

Removed one file.

One removed file is still accessible by collaborators.

I don’t own any of these files. They were shared with me. So what exactly did I just remove then? I can’t possibly remove things that I don’t own? I assume that the last notification indicates that the owner can still access his own file… duh! But the question still remains, what is it that I’m removing then? Removing myself from the access list of a file I didn’t own in the first place?…

Interestingly, most of these files were things like PDF files and shared Google Maps locations, things that were shared publicly, links to which I must have clicked on at some point on different websites. So I was never really “collaborating” with any of these people, the owners of these files, so they never shared the links to these files with me directly and I was never on any kind of access list.