The server is definitely okay with that configuration, for that amount of data even 1 dedicated core is more than sufficient. We have tested on a single core server with about 3 million rows and it was perfectly fine. The bottleneck might be something else.
Have you kept the indexing window open until it finished? While the indexing continues in a backround task a wp-cron job, but that is nowhere near as fast (and it is not as efficient as a regular cron job) as keeping the tab open until it finishes.
If you want, you can add temporary FTP and back-end details, and I will take a performance test and debug.