Oh I just remembered this issue i had. At the end i attributed the wobble to the switches being ever so slightly higher than the stabilizers. Something to consider.
Btw I’ve found that a heavy switch or one with a pronounced bump (i used Bobas) counteracts this wiggle a bit, in the spacebar at least. My reasoning is that they don’t allow that unintentional, non-actuation <1mm depression, which, even when slight, translates to annoying movement on either ends on the stabs.