(necro this old thread)
I am curious, so I begin to dig into the source (
https://github.com/bulletphysics/bullet ... tector.cpp).
First, it uses separation theorem.
Then, it selects the direction (in separation theorem) that yield the least overlapping distance.
After that, it projects a face of body A, and a face of body B - using the direction as Normal. (and B A vice versa)
There might be 6*6 = 36 cases. I am not sure.
Using the projection result, It calls "intersectRectQuad2()" to check whether an overlapping occur.
That would be enough for vertex case.
For the edge vs edge, I believe the same procedure could be used : just use Normal = edge from A (cross) edge from B.
I didn't check it yet.