I see, so you want the variations, but only in one size.
Well, in that case the only possiblity I see is to perhaps exclude all sizes but one- For example exclude M, L, XL and leave only S – therefore all variation colors will appear, but only in that size, so they don’t repeat.