Retrieving face normal from ray result callback

[ajshort]

Hi all,

I'm performing a ray test on a number of triangle meshes and attempting to get the hit face normal in the ray result callback. The way to do this seems to be creating a custom ray result callback and overriding addSingleResult, which gives me access to the local shape info m_triangleIndex member. So far I'm thinking something like:

Code: Select all

btScalar addSingleResult(btCollisionWorld::LocalRayResult &result, bool) override
{
    // ...

    // Get the triangle which generated the hit and calculate its normal.
    const auto *triangleShape = static_cast<const btBvhTriangleMeshShape *>(m_collisionObject->getCollisionShape());
    const auto *triangleMesh = static_cast<const btTriangleIndexVertexArray *>(triangleShape->getMeshInterface());

    const int subPart = result.m_localShapeInfo->m_shapePart;
    const int triangleIndex = result.m_localShapeInfo->m_triangleIndex;

    triangleMesh->getLockedReadOnlyVertexIndexBase(..., subPart);
    // Calculate the face index here.
    triangleMesh->unLockReadOnlyVertexBase(subPart);

    // ...
}

My question is if this is the correct way to do this? Also does anyone have any code for getting a face normal from a triangle index? With all the stride handling and different data types possible the code could become quite complex so I'm hoping there's a simpler way.

[ajshort]

Here's the function I ended up using, let me know if you know of a better way:

Code: Select all

static btVector3 getFaceNormal(const btTriangleIndexVertexArray *mesh, int shapePart, int triangleIndex)
{
  const unsigned char *vertices;
  int numVertices;
  int vertexStride;
  PHY_ScalarType verticesType;

  const unsigned char *indices;
  int indicesStride;
  int numFaces;
  PHY_ScalarType indicesType;

  auto vertex = [&](int vertexIndex) {
    const auto *data = reinterpret_cast<const btScalar *>(vertices + vertexIndex * vertexStride);
    return btVector3(*data, *(data + 1), *(data + 2));
  };

  triangleMesh->getLockedReadOnlyVertexIndexBase(
    &vertices, numVertices, verticesType, vertexStride, &indices, indicesStride, numFaces, indicesType, part
  );

  const auto *index = reinterpret_cast<const int *>(indices + triangleIndex * indicesStride);
  btVector3 va = vertex(*index);
  btVector3 vb = vertex(*(index + 1));
  btVector3 vc = vertex(*(index + 2));
  btVector3 normal = (vb - va).cross(vc - va).normalized();

  triangleMesh->unLockReadOnlyVertexBase(part);

  return normal;
}