the method signature is
static int intersectRectQuad2 (btScalar h[2], btScalar p[8], btScalar ret[16])
But, I paused there and ran through all of the elements.. and it looked okay.. until I tried the fictitious 9th element [8]. Surprisingly it returned the half extent of the box instead of uninitialized memory. You might think that.. so what, I got lucky that it allocated them contiguously.. but.. the method itself uses fictitious element 8.
It starts at the beginning of p, sets q to first element in p using pointer math.
then there are two for loop beginnings.. and it sets pq to the first element of q using pointer math.
then, it loops 5 times.. each time setting pq += 2 (ponter math for moving it two elements forward)
starting with a zero bound;
5*2 = 8
at that point.. pq references the 8th element of the original array p.. which.. is beyond the size of p in the declaration.
So.. either I'm missing something (which could easily be the case), there's either an invalid declaration with btScalar p[8] in the method signature, or there's an overrun.
p is supposed to be the 2d coordinate of the incident face (x, y pairs) according to the documentation..
Code: Select all
// find the four corners of the incident face, in reference-face coordinates
btScalar quad[8]; // 2D coordinate of incident face (x,y pairs)
quad[0] = c1 - k1 - k3;
quad[1] = c2 - k2 - k4;
quad[2] = c1 - k1 + k3;
quad[3] = c2 - k2 + k4;
quad[4] = c1 + k1 + k3;
quad[5] = c2 + k2 + k4;
quad[6] = c1 + k1 - k3;
quad[7] = c2 + k2 - k4;
// find the size of the reference face
btScalar rect[2];
rect[0] = Sa[code1];
rect[1] = Sa[code2];
// intersect the incident and reference faces
btScalar ret[16];
int n = intersectRectQuad2 (rect,quad,ret);
