Thanks for the response!
I have excluded there as well.
However I realized what the problem is.
When it hovers, it finds COMMENTS made in those excluded categories.
I do need comments to be searchable in general – it's quite important. But the comments in hidden categories need to be excluded.
Any way to fix this?