Great!
The fieldsets have classes based on the field or taxonomy names, so you can actually target the fieldset as well as the legend element, like so:
/* For taxonomy term filters */
.asp_filter_tax_{taxonomy_name} {
}
.asp_filter_tax_{taxonomy_name} legend {
}
/* For custom field filters */
.asp_filter_cf_{field_name} {
}
.asp_filter_cf_{field_name} legend {
}Just replace the {taxonomy_name} and the {field_name} in the above code to the corresponding taxonomy or custom field names.