Ok, at the end of the day I'm trying to check if the rotations of two vectors are close.

The rotations I've applied to the two vectors are 5 degrees different.
degreesDifference ends up being 3.53501773.  I sure thought this would be 5.

It should come as no great suprize that I'm missing something super obvious but any help would sure be appreciated.

Thanks,
Dan

3.535... is the correct answer, I checked it in a CAD program.  It has something to do with you rotating the vectors around an axis that isn't perpendicular to them.  If you project the vectors to the XY plane (to which the rotation axis is perpendicular), there will be an angle of 5 degrees between them.  Perhaps someone else can give a better explanation.

Yup... your right!  When I change the rotation matrix to be on the X axix it does come out to be very near 5 (4.99999).  Thanks for pointing that out!

Hmmm... If anyone can provide an explanation for this I could sure use it :)

Imagine they are both pointing straight down the Z axis, rotating them about Z will not separate them.  The more you elevate them off Z, the closer they come to being perp. to Z, the more the rotations will separate them.  Little things like that make corner molding hell to cut.  Not everything comes out to be 45deg like you'd initially intuit.

Maybe I'm going about this all wrong.  Should I realy be comparing the rotation matrix's themselves instead of going all the way to the vector level?  The two objects each have a quaternion stored with their current rotations.  I'm staring to think maybe you would get each axis's rotation from the quaternion and do an angle test on each axis for the comparison.  Does that make sense?  Is there a better way? 

What are you actually trying to achieve here? ie. what's your end goal of this computation?

I've got two puzzle pieces that can be rotated an arbitrary amount along say the z axis.  Piece one the player controls and you are trying to match the rotation of the other piece.  This is happening in 3D as well but I've got the part working to make sure that both pieces are actualy facing the same direction.  Verifying that they are rotated within 5 degrees of each other is where I'm struggling a bit.  I'm probably missing something quite obvious :)

Rotating 5 degrees in 2D will give you a 5 degree difference, but after you normalize the vector you end up with 5 degrees over the square root of 2.

Just like a triangle with 2 sides of a length of 1 gives a hypotenuse of 1/sqrt(2), this will give you 5/sqrt(2), or 3.5355333906f

My guess is that if you didn't normalize them you might get 5 degrees from your acos(dot product), rather than 5/sqrt(2).

EDIT:
If you want to see if they're facing the same way, just use their forward vectors from their rotation matrix, and a dot product. If the dot products are fairly close to 1.0f then they're facing the same direction.

Thanks for the explanation about how the current return value is valid.  That helps a ton.  You mentioned not normalizing but if I don't, it NaN's out.

If you use Acos with values less than -1.0f, or greater than 1.0f, you'll get NaN as a return.

The dot product likely is out of range.

Well, so I'm now setting up my initial vectors to be Vector3.Up + Vector3.Forward + Vector3.Right then applying the rotations via the transform method.  This seems to get enough axis' involved so that using the acos(dot) method returns a value close enough to 5 consistanly that it looks like I can use it.

Thanks for the help!  I've learned more about math and game principles in general here on these boards that I have in 10 years of net surfing.

I've been posting a bit of vector math stuff on XNAWiki. You might check on this page:

http://xnawiki.com/index.php?title=Vector_Math

The second example is "Angle Between Two Vectors", which might be particularly helpful in this situation you've described.