This sounds like a server performance fluctuation to me. I cannot access the url anymore to check right now, but I feel I’m going to get a different results again. It proves the fact that first only firefox felt slower, now (after 2 weeks) both browsers – which should not affect the performance at all.
Please read through these performance related suggestions in the documentation: https://goo.gl/Rn6lG6
Those are very important, especially if you are on a shared host and using lots of plugins requireing lots of CPU. The overall plugin performance more-less corresponds with the server performance, except if there is a conflict or an unknown error.
You can also check your site performance with the P3 performance profiler after, to see if there is any other plugin or theme eating resources.
If nothing changes anything, feel free to post log-in and FTP details to your test environment, and I will debug through the search code to search for possible conflicts or errors.