Calls to btRigidBody::applyForce() accumulate in a data member called btRigidBody::m_totalForce. That value is used in each substep until the end of the full step, at which point it is cleared.
In other words: You MUST applyForce() before the stepSimulation(). That is the only way it works. The force only has effect for the next step, you must re-apply it for subsequent steps if you want its continued effect.
For completeness, and in case any future readers are wondering: you can achieve a perpetual "force" that survives multiple steps by setting a per-object gravity: btRigidBody::setGravity(). Yes, it is actually an acceleration, however that makes it even easier to tune and you if necessary you can always divide your "force" by mass to compute the acceleration to use.
I looked around the code hunting for the best way to do it. I could think of various ways to do it, each with tradeoffs (mostly CPU cost vs correctness) and it wasn't quite clear to me how you would choose to do it because I don't know why exactly you want capped velocities. For example: is it just to reduce likelihood of tunnelling? Would it be applied to ALL object or do you have particular objects which, for game-logic reasons, should never exceed some velocity?
Assuming you need it for select objects the simplest method that occurs to me would be to implement a SpeedLimitAction from btActionInterface. You would override its pure virtual method updateAction() to measure the velocities of the body it cares about and clamp them as necessary. For each object which requires the clamped behavior you would instantiate a SpeedLimitAction (giving it a pointer to its object) and add it to the world with btDiscreteDynamicsWorld::addAction(). The Bullet API is guaranteed to call SpeedLimitAction::updateAction() every substep after all the collisions and integrating the transforms.
Note that the actions are updated after the object has actually moved, and as a consequence of collisions, which means your object might effectively move for the frame with higher velocities than you want: the clamping would happen after the fact.
One drawback of the action method is the action will be called even when the object on which it would act is inactive. To avoid this problem you could instead give it a CustomMostionState where you have overridden the btMotionState::setWorldTransform() to do the velocity measurement and clamping. Bullet will call setWorldTransform() on the MotionState every substep, but only when the body is active.
Hrm... you could also just create one Action which knows about all objects it should be clamping, and it could loop over them all, check if active, and then clamp if necessary.
If you really need to clamp its true and effective velocity before it actually moves, then one way could be to derive a CustomDiscreteDynamicsWorld and override the virtual method btDiscreteDynamicsWorld::integrateTransforms() to loop over all the objects to be clamped before calling the base class method. That method is called after collisions and constraints are solved.
Edit: Whoops I accidentally posted this before I was done.
