I understand completely – it is very hard to create a tool that works for all use-cases, so you’ve chosen certain ones over others. I do think options for choosing how to handle each type of punctuation would be useful, and Relevanssi’s code seems like a good starting point. Someone who wants to index “AB:123-44_42;567” could do so, and I could remove all of that. One of the many reasons I chose ASP is for its enormous amount of customizability, and this would be another example of that.
However, the relevant question for you is “would many other ASP users sufficiently appreciate this to make it worth your time?” I don’t know.
But, don’t take me into consideration for this decision. Regardless of how you proceed, I plan to replace ASP’s tokenization with a python NLP package (spaCy) – as a world-class NLP tool, I assume it does an excellent job with tokenization, but more importantly, it produces lemmas that should improve my table size and search performance considerably. It also creates all sorts of other interesting and useful data – identification of Parts of Speech, Named Entities, and more. Moreover, I’m also just curious to tinker with it all.
Unlike with Apache Tika, I really doubt this is something that would ever be worth you working to integrate with ASP, but I figured I’d mention it.
The extensive hooks in ASP make all of this relatively easy to do, and I’m sure you won’t mind adding some others should they be necessary.