My angle constraint is: C = (atan2(n2) - atan2(n1)) - theta = 0
This is in 2D, so it's constraining two linesegs which share a vertex (instead of two triangles which share an edge), and the normals n1,n2 are the perp()'d lineseg vectors (instead of the crossproduct of triangle edges).
Actually, I started out using the perp'd lineseg vectors, but soon realized that it was simpler to just use the lineseg vectors themselves. I'm not sure if you could do something analogous in 3D.
This angle constraint behaves _really_ well (though it may be more due to the solver than the formulation of this particular constraint), you can set it to be perfectly rigid (turning the two linesegs into a single rigid shape).. it's much better than anything else I've tried. Start in any arbitrary configuration and they'll snap together regardless of the initial error!
The Jacobian is (found via LiveMath):
points p1,p2,p3
let vA = p1-p2, vB = p3-p2
constraint is C = atan2(vB) - atan2(vA) - theta
dC/dp1 = Perp(vA) * ( -1 / ( ((vA.y/vA.x)^2 + 1)*(vA.x^2) ) )
dC/dp3 = Perp(vB) * ( -1 / ( ((vB.y/vB.x)^2 + 1)*(vB.x^2) ) )
dC/dp2 = -(dC/dp1) - (dC/dp3)
It's possible that the denominator terms on the right can be simplified/reduced to vector operations -- the program I'm using to derive the Jacobians prefers scalars, so I've been treating each vector component as a seperate scalar, it's a bit awkward
The only problem is that when p2.x == p1.x or p2.x == p3.x, things are undefined; currently I'm simply skipping/ignoring the constraint when this happens (similar to when a distance constraint finds a current length of 0, thus the direction of projection is undefined).
This hasn't caused any problems so far, it may simply be due to the fact that the program I'm using only supports arctan(y/x), which is undefined for the same cases. Frankly I have very little theoretical understanding of trig functions or the difference between the implementation of atan() and atan2().
It seems to me that there should be a simple fix to this problem: rotate everything 90deg, solve in that space, then rotate the solutions back. Haven't tried this yet though, as so far the problem case doesn't happen much in practice (and skipping the constraint in those cases seems to "work").
Just to be sure, atan2(v) = atan2(v.y,v.x)
Finally, would it be possible to reduce your 3D constraint into 2D? It seems like you could simply use the shared edge between triangles as the normal of a plane, and constrain the angles in that plane. I guess the problem is how to distribute the correction between the two vertices which form the shared edge..
raigan