Hi,
Thank you for the details!
Error 504 simply means that the server probably got exhausted, and did a temporary reset on the PHP process. I see you have a decent number of posts, and a very large number of possible keywords. This will take some time to index, and if the server does other things as well, it can get exhausted.
While the plugin checks after each iteration the request times, perfrormance etc.. and adjusts the next iteration according to that, even waits between iterations to cool off the server a bit – it is still possible the server cannot handle that.
If you encounter a 504 error during the indexing, especially during the end, then wait a few minutes, and then use the continue index button. Eventually it will finish.
If the server exhausts a lot due to the large data volume, then maybe switching to a different server would be recommended.
I am testing the index table at this moment (did around 14%), and the server seems to work okay, indexing is fast (relative to the data size). I think it just struggles from time to time, but probably will be able to handle it.
Try continuing the existing index, I increased the number of posts per iteration, it might help the server to fork fewer number of processes.