document-set filters?

I have a library with multiple document sets, each of which has custom metadata. I want to create a filter for use within doc-set A that filters on A’s custom metadata, and a filter for use within doc-set B that filters on B’s unique metadata, etc.

So far I see how to create a filter at the library level but not at the doc-set level. This is a problem because B’s metadata displays in the filter panel even when in A. How do I create filters unique to each doc set?