Ok so it seems I was missing a step.
The worldViewProj matrix will not project to the screen space but to the camera's projection space.So in fact I think what we have is this:
Out WorldViewProj In(X') (m11, m12, m13, m14) (X)(Y') = (m21, m22, m23, m24) (Y)(W') (m31, m32, m33, m34) (Z)(1) (m41, m42, m43, ...