Having multiple instances of the keywords in the index table is perfectly normal, as the unique key is not the post ID, but multiple column combinations, including the id and the primary term. If you don’t mind, I won’t go into details why or how exactly, the main reason is – this is how indexed queries work the most efficiently. Relevanssi uses a very similar approach in their index table.
The local php PDF indexing tools are of course much less efficient and will not be able to extract as much information. Making some modifications to the tokenize process on your end may fix problems for you, but could cause issues for dozens of other customers – especially regarding PDF contents. Trust me, we have looked at more than the Relevanssi code to make that function as good as possible. Lots of tests, lots of trial & error goes into that method – and of course it may never be perfect, but we need to consider compatibility with all kinds of different configurations.
Of course if you have any specific suggestions, I am more than happy to take note and incorporate them. If they pass the tests, I am surely adding them to the final release.