Yeah, term meta is great, but unfortunately devs are not using it yet. I was hoping advanced custom fields plugin will switch first, but last time I checked, it is still using a different method storing term meta (in the options table).
I see you have lots of different cases, so here is a very generic filter code I put together, which helps you distinguish different result types:
This should answer all your questions and should cover all possible cases.