Oh okay – in that case the categories are actually not the same. Some translation methods create new category objects, and new post type object. If you selected the category groups manually, then that explains the issue – those categories (as objects) actually do not exist on the japanese version.
The best way to get by this I believe would be using 2 search instances. Try configuring the first on the english, the other on the japanese back-end. If the back-end is changed to japanese, hopefully the translated categories will show up, then allowing the grouping by those categories.