Because the search queries are the only parts, which can be indexed fully. Filtering needs to make some table joins – it is not possible to index the combinations – it would lead to exponentially huge database (combination per keyword), probably tens or hundreds of gigabytes for even a couple of thousand products. For 5000 products each with at least 250 fields (and there are probably much more, because of revisions, and WP internal fields etc..), that can grow to a huge table join.
I tried a few combinations, but worst case scenario I was getting ~4.3 second requet times: https://i.imgur.com/Rhs9413.png
Try running an “OPTIMIZE TABLE” command on the wp_postmeta table, to make sure everything is arranged there as well. It sometimes does speed up the queries by a lot.
Although the 4 second response time seems about realistic to me because of the huge number of fields per post.