My current code is based entirely on the BasicDemo demo (as I said, at the moment I'm just playing around). What I've been trying to do is make a little car, and it seems to work well enough (although even with small wheels there seems to be a bit of a "bounce"problem), however when I make the wheels a bit bigger, I get bad results.
So here's the problem : I have a big box collision shape which serves as my ground (the one from BasicDemo, I haven't changed it) and cylinders welded to a convex hull which are my car's wheels (for the record, I also tried with a box instead of a convex hull, so that's probably not the problem), and sometimes the wheels partially go through the ground before bouncing back up.
Watching the bug with AABB rendering on shows that the collision between the ground and the wheel's AABBs works well; the problem seems to stem from the AABB itself, as it sometimes becomes smaller than the actual collision shape, thus allowing bad collisions to happen.
Here are screenshots to illustrate :
(the box shouldn't be allowed to pass right through that cylinder)
(note that in this screenshot the cylinder isn't even being used as a wheel)
http://img829.imageshack.us/img829/7855/aabb.png
http://smealum.net/aabb2.png
I also uploaded a (windows) executable : http://smealum.net/bullet_test.zip
So I don't know what the problem is here; I guess I might just be doing something wrong, and I'd like to know what. Either that or there's a big flaw in the engine, and I guess it would have been caught by now so I figure my code is the problem.
EDIT : I seem to have found my problem, and it was indeed a very stupid mistake on my part !
For those curious, my c->wheelSize btVector3 had wrong values; I didn't realize that both the Y and Z values had to be set to the radius, and so that broke things a bit (I'm not sure if it actually broke the physics or just the demo rendering). At any rate though, with the right values everything seems to work just fine now.