HI,
Thank you for the details, it helps me a lot!
The performance issue might be related to a server issue as well. I tried to access the site back-end, but it takes me over 2 minutes to simply load a page. The configuration was perfectly fine.
I have turned off the custom ajax handler, and it should be much better now – around 1-3 seconds. I am however seeing simple empty WP maintenance requests taking over 10 seconds to respond, so there might be a server performance issue as well, I recommend checking that out.