Okay, I have found the issue, it is however not related to the search.
The filters were working fine, which I found strange, so I started checking the actual values of the publish_date field and found the culprit.
The issue is that some values of that field are stored as timestamps and some values are stored as ACF dates. If you take a look at these debug outputs, in this one the ACF date values are stored, but on other entries (I guess older entries) the timestamp is stored.
I suspect this field was imported and the data types are not matching with the current ACF date type. The only way to resolve this is to convert the incorrect timestamp data to ACF date data type. (especially if you want to use ACF date field) For display purposes it’s fine, ACF and Ajax Search Pro can both convert them, but for searching it is a massive issue, as there is no way to query on both at the same time, the data types must be the same.