Everything got tested with different video driver versions and also on
the REF device, always behave the same way. DX10/C++/HLSL.
I capture a frame in PIX, then go to the last draw call, then I select
a particular pixel in the Render tab of PIX, right click then select
"debug this pixel". It's history simply contain Initial > Clear >
Draw > Final color. The pixel shader ouput of the draw is equal to
float4(0.494, 0.282, 0.004, 1.000). If I click "Debug Pixel" on this
page, then step all the way until the end, and check the value returned
by the pixel shader, it show float4(0.193, 0.113, 0.003, 1.000). This
is supposed to be the correct expected value, proving that my code
actually seems ok, but unfortunately the color displayed is the first
one mentioned earlier which makes no sense.
Another strange thing. Debugging this way with PIX show me that a
certain variable, determined by a series of "if" (see below) is set to
the expected value. If I force the variable in the shader code to this
value, of course it mix up things for other pixels, but for this one it
fix the problem and return, and most importantly display, the correct
color(0.193, 0.113, 0.003, 1.000). Debugging one way or another (exact
same frame) in PIX show the exact same values all the way for every
variables of the whole pixel shader, except for the final color
displayed on screen.
The bug happens only for certain pixels when two float values are
nearly equals in the "if" series talked earlier, so that's why I had
the idea to force this specific variable, and was amazed that it
changed something. The variable in question is "cubeMapNum" in this
code (based off an nvidia sdk sample)
float maxCoord = max( abs( DirToLight.x ), max( abs( DirToLight.y ), abs( DirToLight.z ) ) );
[flatten]
if( maxCoord == abs( DirToLight.x ) )
cubeMapNum = DirToLight.x > 0 ? 0 : 1;
[flatten]
if( maxCoord == abs( DirToLight.y ) )
cubeMapNum = DirToLight.y > 0 ? 5 : 4;
[flatten]
if( maxCoord == abs( DirToLight.z ) )
cubeMapNum = DirToLight.z > 0 ? 2 : 3;
uint shadowMapIndex = cubeMapNum+(l*6);
So in fact, PIX step-trough the shader return the correct value, but
D3D don't display this one. If anyone have any idea of what's going on
please reply. I guess this have something to do with shaders
optimizations or numerical instability, but as I said, the result is
the same with any driver, device, debug or release. When in debug,
shader is compiled with D3D10_SHADER_DEBUG and
D3D10_SHADER_SKIP_OPTIMIZATION flags.
