I wrote an issue about this a while ago, and just wanted to bring it to attention as it's turning up all the time in my current project.
When calculating the angle between two quaternions and the angle is very small, btQuaternion::angle does in some cases return nan. It is because of numerical precision issues when calculating a square root. The issue and explanation including a proposed fix is here:
http://code.google.com/p/bullet/issues/detail?id=237
I run into this quite often when using the btQuaternion's slerp function to smoothly play back some aircraft's flight recorder data. It happens when interpolating between two very similar quaternions.
As written in the issue, btVector3::angle will potentially have the same problem so I would recommend fixing that one too while at it.
I'll make a patch if necessary.
Best regards,
Ola
