The moment I enter a search query and select a filter for a certain post type, the search results also return attachments. I don’t want to show the attachments when they are not relevant to a filter
Try turning off this option here: https://i.imgur.com/qErO3wI.png
By default media files does not have taxonomies connected to them, but this may resolve the issue.
This may not be possible then I’m afraid. If you want I can try experimenting with the options, then please add temporary FTP and back-end access and I will take a look.