I'm looking at some issues arising from using a zero collision margin. I know, not really supported, but I'm coming across some failure cases that shouldn't really be failing regardless.
I think I've narrowed it down to the interaction between GJK and EPA in some marginal cases. The EPA part can return several failure codes, but btGjkEpaSolver2::Penetration() only checks for the generic EPA::eStatus::Failed code. That does not catch EPA::eStatus::InvalidHull which can happen in some borderline floating point precision cases.
GJK then ends up using the EPA result even though it isn't really valid. As a result I can get contact normals being flipped around (btGjkPairDetector::getClosestPointsNonVirtual() ignores the normal returned by the penetration solver and uses the vector between the contact points instead, which are incorrect in the invalid hull case).
Is there reason to believe that this will never happen with a reasonable collision margin?
---JvdL---