Hi Attila,
Unfortunately the index table only works with post type contents. The users database is indexed fairly well in general. For the maximum possible performance try selecting the least amount fields to search.
For that amount of users searching their display names should be more than sufficient to give a fairly big results pool: https://i.imgur.com/1m2cCh5.png
Turning off searching in user meta fields should have a great impact on the performance as well: https://i.imgur.com/kXNTAVL.png
Basically any field you can avoid searching will have an impact on the performance.