Hi,
It happens because WordPress stores the terms and the posts in separate tables, and it is not possible to fetch them across the sites. This is because the terms can have the same ID fields and therefore it would reference a completely different category, or nothing at all if the term with the ID does not exist.
Unfortunately there is no way around this due to this limitation.