Hi!
The performance indeed greatly depends on the server performance, as well as the overall setup.
There are number of things you can do to maximize the possible performance, you can find these tips here: Improving search performance
The custom ajax handler (described in the linked docs) as well as the index table engine can make a real difference.