I want to use bullet physics to measure the min distance between convex objects(btConvexTriangleMeshShape)
That statement does not agree with the image you supplied: some of the objects in the image do not render as convex.
Also, you talk about whether "point is within shape" but your image has text about "point is not within model". Are you aware that there would tend to be a difference between the convex shape approximation of a concave model? Perhaps you should use better approximate shapes, like a really tight btCompoundShape around all of the convex parts of your concave model.
btConvexTriangleMeshShape derives from btConvexShape which introduces the set/getMargin() API. I haven't looked at the code but it is possible there is a default collision margin at play for btConvexTriangleMeshShape and the "nearest approach" algorithms may be supplying points on the surface with margin. Dunno if this is the case, just saying it is a possibility that you might want to verify.