Thank you for the proper details, it helps me a lot!
It was mostly due to a recenlty discovered bug, that is related to the latest version of the plugin, when using together with WooCommerce. It does not affect all installations though. It was related to stock based filtering. I have disabled the problematic line in the plugin files, now it should be much better.
While I was there, I've also configured and enabled the index table engine for you to give it more performance boost. According to the performance tracker now it's around 10-20x times more responsive. More about the index table here.
I've also tried to enable the custom ajax handler, but a security plugin is not allowing access to it. In case you want to use it, you will have to whitelist the following file or URL somewhere:
There is also a chapter in the documentation about performance improvements, if you are interested, you can read it here: Fine tuning the search performance
I hope this helps!Best,
If you like my products, don't forget to rate them on codecanyon :)