Modern Tessellation Based Planet Renderer needs Collision

[PaloDeQueso]

So I wrote a basic planet shader which takes a very low poly (32 triangles) sphere and tessellates it to up to 2 million triangles of detail for the surface of the planet. This is a real time dynamic LOD all done with the tessellation processor on the GPU. That means I never store the result. As far as the other parts of the shaders are concerned, as well as the c++ end, it's still a 32 triangle sphere. The issue comes when trying to simulate physics. I've already worked out how to do gravity when near planets, that's fine. What I'm worried about now is how to simulate collision when something actually lands on a planet.

Obviously, it's out of the question to generate a 2 million triangle shape and pump that into bullet. But the planet is generated from perlin noise based height maps (using cube mapping), so I thought maybe I could use the height map shape in bullet, but I realize that's only for 2d. I can certainly generate some very basic geometry for bullet but because of the size of a planet, I'm worried it won't be high enough resolution when a player is just walking on the ground.

So my question is, am I still stuck generating LODs for physics simulation anyway? Or does anyone who perhaps has more bullet experience have a better idea of how to accomplish this?

I am still working on the atmospheric scattering shader, so I haven't delved into this but am starting to think toward the future a bit, and can't lie that I'm a bit worried that it will be a show stopper.