Yes, I more or less understand how your index works and it is extremely similar to Relevanssi. I have no problem with the methodology. Its just a mystery that certain words are not working for partial matching. I’ll reassess when I’ve made the changes that I am looking to incorporate.
My comments on the tokenization were just an aside to explain what I’m doing, but here are a few modifications that I made before starting to explore using an NLP package. They made a major difference in reducing junk entries – many thousands of entries full of punctuation were removed (ones which mostly do not exist in the Relevanssi index).
// Replace end-of-line hyphenation with nothing, to create a full word. Needs to be before the line for removing line breaks.
$str = str_replace( array( “-\n”, “-\r”), “”, $str );
// Replace hyphens, and em and en dashes with space.
$str = str_replace( array( “—”, “–”, “-“), ” “, $str );
These were added to the array for various symbols.
“: “,
“__”,
//quote replacement
“’”,
“‘”,
“’”,
“‘”,
“””,
““”,
“„”,
“´”,
“″”,
I particularly like the user customizability options of Relevanssi for which things to replace and with what (see attached screenshot). And their TAIKASANA magic method for preserving decimals, ampersands and hyphens/dashes while removing all other punctuation with
$a = preg_replace( ‘/[[:punct:]]+/u’, apply_filters( ‘relevanssi_default_punctuation_replacement’, ‘ ‘ ), $a );
is quite clever.
Anyway, this is all beside the point of this ticket. If you test and incorporate them, I would be very surprised if they created any negative side-effects for anyone, especially if combined with user-selectable options in the backend. I hope this has been helpful.
I will update the ticket when I have made my changes with the NLP tool.