Basically I am making a RenderTarget which is meant to be a transparent texture, then rendering various images on it (including text) with alpha blending using the standard:

device.RenderState.SourceBlend = Blend.SourceAlpha;
device.RenderState.DestinationBlend = Blend.InverseSourceAlpha;
device.RenderState.BlendFunction = BlendFunction.Add;

The problem I'm running into is that this standard method of blending assumes that the destination is opaque.  Even though all the data in the RenderTarget texture has alpha set to 0, it still has a color associated with it and that color gets taken into account when blend operations are being performed on what I wanted to be a transparent texture.

I know that I could just turn off alpha blending and not have that problem when rendering on the transparent portions, but I also want to render using alpha blending over portions that have already been written to.  I don't see any easy way to take into account both the SourceAlpha and DestinationAlpha with only Add, Subtract, Min and Max functions to work with in the renderstate.

I imagine I could make a pixel shader to take care of this, but I've never actually written one yet and it seems like there should be a simpler, less processor intensive way to take care of this.

I found the blog post you mentioned: http://home.comcast.net/~tom_forsyth/blog.wiki.html#%5B%5BPremultiplied%20alpha%5D%5D

At first glance it seemed the answer to all my problems, but after I read through it very carefully a few times and I don't see how the claims he makes match up with the methods he is describing.  From what I gather he is proposing packing all textures in an alpha premultipled state.  Given the normally accepted alpha blending technique equation:

S.A * S.RGB + (1 - S.A) * D.RGB

It looks like the premultiplying only saves the step of S.A * S.RGB giving you a little extra performance.  Maybe I am just missing something obvious, but I don't see how that would fix the problem of other alpha = 0 colors bleeding into the partially visible colors around it when they are sampled for scaling operations.  I also don't see how it fixes the problem I'm dealing with when compositing partially transparent textures.  It claims to fix both of those problems though.  It seems to me that when composing multiple partially transparent textures, the only way to handle it is to take into account the alpha of the pixel being drawn over.

I played around with pixel shaders for a bit, but couldn't figure out an easy way to sample the source texel and destination texel and then blend the two myself.  It seems like it should be possible, but all the examples I saw only seemed to work using input textures.

After some thought, I did think of a way to fix the problem using a 2 pass sprite rendering technique that uses an equation I thought up to accurately take into account the alpha of the destination rather than just the source:

S.A * S.RGB + (1 - S.A) * (D.A * D.RGB + (1 - D.A) * S.RGB)

Basically I run the first pass (D.A * D.RGB + (1 - D.A) * S.RGB) with the these settings:

renderState.DestinationBlend = Blend.DestinationAlpha;
renderState.SourceBlend = Blend.InverseDestinationAlpha;
renderState.BlendFunction = BlendFunction.Add;
renderState.AlphaDestinationBlend = Blend.DestinationAlpha;
renderState.AlphaSourceBlend = Blend.InverseDestinationAlpha;
renderState.AlphaBlendOperation = BlendFunction.Add;
renderState.AlphaFunction = CompareFunction.Always;

And then I render the second pass (S.A * S.RGB + (1 - S.A) * D.RGB) with the these settings:

renderState.DestinationBlend = Blend.InverseSourceAlpha;
renderState.SourceBlend = Blend.SourceAlpha;
renderState.BlendFunction = BlendFunction.Add;
renderState.AlphaDestinationBlend = Blend.InverseSourceAlpha;
renderState.AlphaSourceBlend = Blend.SourceAlpha;
renderState.AlphaBlendOperation = BlendFunction.Add;
renderState.AlphaFunction = CompareFunction.Always;

This produces the effect I am looking for, but it annoys me that I have to render all the sprites twice.  It is all cached on a texture so it is not much of a performance hit, but I would really like to know if there is some magic to premultiplied alpha that I'm missing or a simple pixel shader that could accomplish this with one sprite rendering.  Any help would be appreciated.

Jesse:

It looks like the premultiplying only saves the step of S.A * S.RGB giving you a little extra performance.  Maybe I am just missing something obvious, but I don't see how that would fix the problem of other alpha = 0 colors bleeding into the partially visible colors around it when they are sampled for scaling operations.  I also don't see how it fixes the problem I'm dealing with when compositing partially transparent textures.  It claims to fix both of those problems though. 

I'm feeling like an idiot here.  Usually I just need to be pointed in the right direction and I can run with it pretty well, but this still isn't making complete sense to me.  I think I have a better idea how it would work, but now I'm slightly hung up on how to implement this all in XNA practical use.

This is my understanding of how you would implement it: You need to start with textures in an alpha premultiplied state, then you can render it onto another texture in a premultiplied state (and repeat as many times as necessary) and then you can render it onto the backbuffer and it will all be nicely blended?  Do you keep the same settings for all of these transitions?

renderState.DestinationBlend = Blend.InverseSourceAlpha;
renderState.SourceBlend = Blend.One;
renderState.BlendFunction = BlendFunction.Add;
renderState.AlphaDestinationBlend = Blend.InverseSourceAlpha;
renderState.AlphaSourceBlend = Blend.One;
renderState.AlphaBlendOperation = BlendFunction.Add;

Are there any other settings I'm missing in the implementation or should it work like that?  I'm still trying to wrap my head around the process.

I haven't tested that mehod yet, I was just making sure I had the concept right.

Why the SeparateAlphaBlendEnabled = true?  After reading the documentation, it doesn't look like I even need the lines of code specifying the alpha channel blending since by default it just uses the same blending as specified in SourceBlend and DestinationBlend.  Unless I'm misunderstanding something with premultiplied alpha, you apply the same blending to alpha channel as well, right?

That's right.

No need for SeparateAlphaBlendEnabled. Just make sure to clear your render target to 0s before starting compositing.

It works.

I have it working with my composition shader ( which basically composes multiple texture layers into a single alpha texture and then uses this texture in the final rendering pass writing to the framebuffer)

This sort of technique is used in Company Of Heroes. If you look at the screenshot here http://www.nathanm.com/images/blog/CompanyOfHeroes.jpg ... all these splats on the ground ( junk, tire tracks, dirt patches  etc) are a separate alpha based texture which is composed separately out of multiple transparent decals and then overlayed on top of multutextured terrain.