Oh, okay, I think I understand now. You need both products and taxonomy terms as results (brands), but not some of the brands. Based on your screenshot, your configuration is okay, and to exclude some brands, you will have to use the box on the General Options -> Sources 2 panel: https://imgur.com/bfvBBZq
It’s less convenient, as you need to type in the brand IDs, but this will not affect the product results in any way, only the taxonomy term results.