Yes – that is intended with that option.
Normally this is not needed, but when the search is within a fixed/absolute positioned placeholder, which auto-hides on scrolling or other actions, the fixed search results container have nowhere to move and they glitch out to the sidebar. This option is the only way to deal with it, as it is nearly impossible to tell if the actual search regained visibility and if the results should be visible again.
You can also try switching the results position to block instead of hovering, and turn this option off, that will prevent the glitch as well, but I am not sure if that layout is what you are looking for.