Yeah, that sounds plausible. I don't really know much about the internals of Physics simulations (I leave that for the clever people

Anyway, I've tweaked the Dynamica plugin to get a similar behaviour from it (what I'm working on is a Maya plugin too, so it seemed the easier option). I've added a "fixedTimeStep" attribute to the solver node (it shows up in the Extra Attributes section of the Attribute Editor), allowing the third parameter to the stepSimulation function to be changed interactively, and an "interpolate" attribute, which, when false, means that maxSubSteps is 0. Decreasing the fixed time step value, or switching "interpolate" off and increasing the substeps, causes the twitchy behaviour.
I'm wondering if perhaps it is something to do with the way the timing happens in Maya, as reducing the time steps to equivalent values in the demo code I hacked before worked OK (when I set maxSubSteps to something sensible!).
I've attached the modified Dynamica code and a test Maya scene (for Maya 2009). I've also attached, separately, some fixes I had to do to get the plugin to compile (though it could be just our build system being finnicky that meant it didn't work for me as-is).
Thanks,
Kate