Oh, I see the problem now. The cache is not automatically cleared after changing the priorities. I recall disabling that intentionally for some reason. I forgot to add this to the documentation.
If you clear the cache on the Cache Settings submenu after making the changes on with the priorities, the results will appear in the correct order.