Hi!
It’s determined based on which of the categories/terms are excluded from the search. If a term is excluded, it will appear as unchecked. You can exclude them on the Advanced Options panel: http://i.imgur.com/EOnJPq2.png
There are similar exclusion boxes on the Frontend search settings panel, but those only determine which of them should be visible at all.