The unexpected results are probably because you are not applying the math right. If you apply the math wrong with matrices, it'll be as wrong as with quaternions.
In general, for the kind of operation you want, figure out what the order of operations is. Then multiply the matrices in question from left to right to apply the order.
First, you want to rotate the camera around the up axis for yaw.
Then you want to move the camera back (in world space) for distance.
Then you want to rotate the camera around the X axis (in world space) for elevation.
Now, "moving the camera" means "moving the vertices the other way." Hence, this is a reasonable approximation of what you want:
Matrix View =
Matrix.CreateRotationY(-camYaw) *
Matrix.CreateTranslation(0, 0, -camPos) *
Matrix.CreateRotationX(-camElevation);
If you're currently trying to do this using quaternions, you should know that in XNA, quaternions multiply right-to-left, whereas matrices and vertices multiply left-to-right. Very confusing.
Jon Watte, Direct3D MVP
kW X-port 3ds Max .X exporter
kW Animation source code