I want to use alpha objects as shadow caster in shadow map algorithm. for example leaves of trees that sunshine can across from them.
I try two ways.

 

1: at pixel shader i fetch alpha of texture  and check if alpha value is less than some value i pass white color to avoid generating shadow

return (tex2D(alphaDiffuseSampler, In.Texcoord).a > 0.6)? In.Depth : 1;

but because of zbuffer this cause that shadow of underneath objects waste.

 

2 : i try to fetch alpha in vertexshader 3 to change zbuffer but i can't fetch alpha in vertex shader 3 .I use textureformat color and point sampling but always i get zero value. 

if (tex2Dlod(alphaDiffuseSampler, float4(In.Texcoord,0,0)).a <= 0.6)
    {
        Out.Depth = 1;
        Out.Position.w = 1;
    }

which way is better and how?

When you say alpha, do you mean alpha blended (where you can have fractional values that make things partially-solid) or alpha tested? (where the alpha channel just cuts out some pixels entirely, while others are drawn solid).

If you really mean alpha blending, this is not possible using conventional shadow maps. A shadow map can only store a value, or not a value - there's no way for it to store a partially blended fractional value.

If you mean alpha testing, you just have to apply the alpha test when you are drawing the shadow map, and cull any pixels that have zero alpha. The easiest way to do that is by making sure the pixel shader which renders your shadow map outputs the same alpha value as your main scene pixel shader, and then enabling the alpha test renderstates. You could also do this inside the shader by examining the alpha value and using the clip() HLSL intrinsic, but that can be sometimes slower than using the alpha test renderstate.

I mean alpha test. But which SurfaceFormat i must use for shadow map. Can i use SurfaceFormat.Single for RenderTarget for better quality? or other format that has alpha channel like SurfaceFormat.Color.

thanx for your answer.

 

You don't need an alpha channel in your output rendertarget. When you use alpha test, you are simply deciding whether or not to draw each pixel. You can still use a single floating point rendertarget: you just write the depth values for pixels that are solid to it, but not the ones that are transparent. There is no actual alpha blending or need to store the alpha here - it's just a "should I write anything at all?" boolean test.

I mixed up. maybe i am stupid. when i use SurfaceFormat.Color for rendertarget and set AlphaTestEnable to true shadow work correctly.

but when i use SurfaceFormat.Single not work , because i have one red channel and no alpha to test !. Can you show me how setup RenderState and what code in shader?

Alpha testing happens before the output is written to the framebuffer. Even if your framebuffer only contains a single red channel of data, you can still output a float4 color value from the pixel shader, and test the alpha of this value. If the alpha test passes, the red component will then be written to the framebuffer.

struct VS_ShadowAlphaIN
{
     float3 Position : POSITION; 
     float2 Texcoord : TEXCOORD0;
};

struct VS_ShadowAlphaOut
{
     float4 Position : POSITION;
     float2 Texcoord : TEXCOORD0;
     float Depth : TEXCOORD1;
};

VS_ShadowAlphaOut VS_RenderShadowMapAlpha(VS_ShadowAlphaIN In)
{
     VS_ShadowAlphaOut Out = (VS_ShadowAlphaOut)0; 
     float4 posWorld = mul(float4(In.Position,1), world);
     Out.Position = mul(posWorld, viewprojection);  
     Out.Depth = Out.Position.z;
     Out.Texcoord = In.Texcoord; 
     return Out;
}

float4 PS_RenderShadowMapAlpha(VS_ShadowAlphaOut In): COLOR
{
     // *** worked
     //float s = (tex2D(alphaDiffuseSampler, In.Texcoord).a > 0.6)? In.Depth : -1;
     //clip(s);
     //return s;
 
     // *** not worked
     return float4(In.Depth,In.Depth,In.Depth,tex2D(alphaDiffuseSampler, In.Texcoord).a);
}

technique RenderShadowMapAlpha
{
     pass p0
     {      
          CullMode = None;
          AlphaTestEnable = true;
          AlphaFunc = LessEqual;
          AlphaRef = 150;
          VertexShader = compile vs_2_0 VS_RenderShadowMapAlpha();
          PixelShader = compile ps_2_0 PS_RenderShadowMapAlpha();
     }
}

Where is my mistake?

Maybe your card doesn't support pixel processing operations for floating point surface formats? (not all do).

You can check for this using GraphicsAdapter.CheckDeviceFormat with QueryUsages.PostPixelShaderBlending.

sorry for my late

 you are right

Framework.Graphics.GraphicsDevice.CreationParameters.Adapter.CheckDeviceFormat(DeviceType.Hardware,

Framework.Graphics.GraphicsDevice.DisplayMode.Format,

TextureUsage.None, QueryUsages.PostPixelShaderBlending, ResourceType.Texture2D,

SurfaceFormat.Single);  is false

 thanks shawn