I did a few tests, and most of them was under 2 seconds, here you can see it: https://i.imgur.com/lHrtCjR.png
Additionaly here is a gif video of another test, take a look: https://i.imgur.com/fLQ6chc.gif
Every request was under 2 seconds.
Are you sure it wasn't a server outage? What kind of server are you using? Is it apache, nginx? How many websites are running on that server? Is the server load low if you check it via SSH?
The 2 seconds respond time is more than excellent for any wordpress ajax application.Best,
If you like my products, don't forget to rate them on codecanyon :)