Page 1 of 1

Heap Corruption?

Posted: Mon Jul 08, 2019 2:05 am
by mrrolf
I think I'm seeing heap corruption with v2.88 when creating the Dynmaics world.

I first observed heap corruption when initializing the btDbvtBroadphase class when creating the dynamic world. After some debugging, I think it's the m_rayTestStacks member variable in the btDbvtBroadphase class that is causing the issue. Here is some code that shows the problem:

Code: Select all

auto classSize = sizeof(btDbvtBroadphase);		// return 0xB0
auto arrSize = sizeof(btAlignedObjectArray<btDbvtProxy*>);	// returns 0x14
auto rayTestStacks_offset = offsetof(btDbvtBroadphase, m_rayTestStacks);		// return 0x9c

char* mem = (char*)malloc(classSize);
btDbvtBroadphase* c = (btDbvtBroadphase*)mem;
memset(mem, 'x', classSize);
memset(mem + rayTestStacks_offset, 'o', arrSize);
memset(c + 1, '_', classSize);	// yes, this stomps memory, but it's to make the memory display easier to read
Using the MSVC memory debug window, I can view the memory at address 'mem'. It looks like this:

Code: Select all

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxoooo
oooooooooooooooo________________
This seems to makes sense. The 'x's are the first members of the btDbvtBroadphase class, and the 'o's are where the m_rayTestStacks member will go. The '_' is memory that doesn't belong to the btDbvtBroadphase class . Everything seems to be making sense.

Then I call the constructor for btDbvtBroadphase.

Code: Select all

auto* pClass = new(c) btDbvtBroadphase();
Now lets view the memory after this call:

Code: Select all

øo*.........ÿÿÿÿ........xxxx....
.........xxx........ÿÿÿÿ........
xxxx.............xxx............
 ¶u.xxxx........................
................................
.......ooooo@...@...ðß..._______
Whoa! The 10 bytes past the end of 'mem' have changed, but they shouldn't have. Stepping through the constructor, the memory changes inside the btAlignedObjectArray::Init() function. The 'this' pointer when the pc is in the Init() call is offset 0xA8 from the this pointer of. That's definitely more than the 0x9c that it should be, but also less than past the end of the class, 0xB0. Writing to this memory would definitely cause the heap corruption that I initially observed.

I'm running VS2017. This is also the first time I've used Bullet, so it's very probable that I'm doing something incorrectly. I just put the .vcproj files into my solution and then built them. The example applications are running fine, so I'm sure sure what's happening :oops:

Does anyone have any insight to what's going on?

_______
Update: It seems that if I force the alignment of class btAlignedObjectArray to be 16, the issue resolves itself. It seems the default alignment is 4, but something at runtime think it's is 16.

I'm compiling to a Win32 platform with VS2017, v15.8.1

Re: Heap Corruption?

Posted: Mon Jul 15, 2019 3:28 am
by mrrolf
Ok, I was finally able to get to the bottom of the issue.

In the BulletCollision, BulletDynamics and LinearMath .vcproj files it defines 'BT_USE_DOUBLE_PRECISION'. This causes scalars to use doubles instead of floats. In the solution, I have a custom allocator. The outside projects didn't define BT_USE_DOUBLE_PRECISION, so they were expecting the scalars to use floats (not doubles). So the Bullet libs had class sizes that included the scalars as doubles, and the external classes had class sizes that

// in the Bullet libs
sizeof( btVector3 ) == 32


// outside the Bullet libs
sizeof( btVector3 ) == 16.

This was causing the heap corruption. The custom allocator wasn't allocating enough memory for the classes, because externally it was using float precision. When the classes were initializing inside Bullet, it was stomping on memory, which was corrupting the heap.

This was resolved by defining 'BT_USE_DOUBLE_PRECISION at the solution level, so all external code knows Bullet is using doubles.

Re: Heap Corruption?

Posted: Mon Jul 15, 2019 9:37 am
by hyyou
Thank for sharing. It is interesting discovery.

Did you use WinDbg?

Re: Heap Corruption?

Posted: Mon Jul 15, 2019 4:37 pm
by mrrolf
Hopefully this will save someone else some time.

I was solely debugging in Visual Studio.