I did a few changes on the code which I suspected it was slowing down the process. It was just a random shot, so I don't know if it actually helped. Right now it takes around 2-5 (5.36 was the highest measured) seconds for me, which seems normal. If however the load is constantly over 5 seconds, then there is something interfering with the plugin. Some spikes might happen (due to server load, maintenance, etc..), but it should not be consistent.
Let me know how is it working now.
If there are constant issues, you should provide temp ftp access as well, so I can run debugging on the search code to search for the cause.Best,
If you like my products, don't forget to rate them on codecanyon :)