In that case, can you please check if this option is also disabled: https://i.imgur.com/mBbrDCB.png
That will get rid off any results, which are not part of the given taxonomy.
For better search performance you can check this documentation. In your case a recommend using the cache for sure.
Please note, that the search peformance always correlates with the server performance – exceptions are probably misconfigurations on very large databases, but that is not your case. The overall site performance is impacted by a lot of factors, but usually the two major issues are the number/quality of the running plugins and the server peformance. I generally recommend not using more than 10-15 plugins if possible, but this greatly varies on the individual plugin code quality and performances. The search response time can not be faster as the server response time, and using a lot of code will generate a lot of overhead (meaning the actual search process may only take a few milliseconds, but there is many other things the server must also execute). You can verify that via the search performance tracker here. As you can see, the actual time the plugin was searching only takes on average 0.05 seconds, which is superb. This is of course not very accurate, but give or take it still indicates, that the search itself peforms perfectly fine.