float precision issues with QuantizedBvh

[StefanEgo]

Hi,

with a certain asset I'm running into the following assertion in btQuantizedBvh::quantize() while trying to construct a btBvhTriangleMeshShape:
btAssert(point.getZ() <= m_bvhAabbMax.getZ());

m_bvhAabbMax was calculated by iterating over all vertices in the asset.
m_bvhAabbMax.Z() is 277755.41 (original value was 277754.41 - which was clamped to 277755.41 in:
btOptimizedBvh::build() - setQuantizationValues(bvhAabbMin,bvhAabbMax); (where a quantizationMargin is added with a default of 1.0))
the AABB-Z-size in my case is 7538564.0 resulting in a very small quantization value of 0.0086930348
AABBMin.Z() is -7,260,808.5

After the min/max values for the AABB were set the triangles are processed with the QuantizedNodeTriangleCallback.
That actually quantizes each node and stores the value.

When we end up with a node of 277754.41 that value is quantized to:
(277754.41 - -7,260,808.5) * 0.0086930348 => 65,532.897
since we are checking a max-value it's increased by one: 65,533.897
cast to an unsigned short: 65533
and ORed by 1: 65533

now when unquantizing that value again:
65533 / 0.0086930348 => 7,538,564.0
7,538,564. -7,260,808.5 => 277,755.5

and there we end up with an unquantized value which exceeds the max AABB value by 0.09 (277,755.5 > 277755.41) ultimately triggering the assertion.

My current solution is to ensure that the margin used by setQuantizationValues compensates for the possible precision error but I'm wondering if that's the correct/best solution to the problem.

Any thoughts?

Regards,
Stefan

[StefanEgo]

The fix for this issue has been checked into trunk:
http://code.google.com/p/bullet/source/detail?r=2615
http://code.google.com/p/bullet/source/detail?r=2617