Oh you are talking about the actual directions.
Restitution has nothing to do with that obviously.
What will control the direction is the normal or the collision.
Is your ball a sphere shape?
If it is a hull or triangulated ball it will not have smooth normals and is thus bound to bounce in a more erratic way. You should not get random directions though. If so then debug what normals you are actually getting.