Thank you very much for all the information, it was extremely helpful 🙂
Currently it looks like we may open a small shop section for additional optional features – like external apache tika processing server – for users needing that feature, for a reasonable monthly/yearly fee. The plugin has too much features already, and I feel like adding more and more might do worse. Some sort of an add-on package/system could probably resolve that too. I am not much in favor of this, but there is so much overhead in the code already.
Local apache tika execution is not possible via including the package (or using a remote) and installing it, as I suspected. Almost all hosts have execution restrictions for the web folders, and PHP has no/restricted access to the server bin folders. We will however add a feature similar to relevanssi for the user to use it’s own tika server instead of paying for one.
I am currently also reviewing the tokenization process, adding a few more tests for all sorts of gibberish to see how the process handles that. Just yesterday I added at least 50 more special characters to handle, as well a process to detect word start/ending with garbage characters.