Hi!
The search performance mostly depends on the server performance as well as configuration and many other factors. I suggest reading the Search Performance Optimization documentation, that will give you tips to get the best possible performance.
Using the Index Table will also improve the search accuracy and performace as well, I would highly suggest trying that feature.