What I want is:



But right now it will just show both. as a triangle on top of the circle

What I eventually want:



As in I want my users to be able to have multiple panels on the screen.

For the panels, I don't want users drawing outside of them, that includes child panels, so I need a way to draw the combined parent and child masks.

I think I might have to set the stencilpass or stencilfunction or something else, but I don't know what to, and randomly changing doesn't work :P

Thanks for your help in advance!

EDIT: the reason I'm not using viewports is because there can be many differently-shaped panels

It sounds like you want something like this?

Unfortunately, I need something more complex.
I need to combine masks, and If I try to combine it by Texture2D.SetData and .GetData it will be too slow.

I'm pretty sure there is a better way to do it, by setting the stencil funciton to something different, or some other stencil property.

Basically, what I mean by combining the masks, is this:



This is the basic way my GUI Engine is drawn

-Draw Engine
   -Draw Frame
   ^  -Draw Mask
   |   -Draw Self
   |---Draw Children

And when drawing the children's masks, I want them to combine:

Pixel with >0 Alpha + Pixel with >0 Alpha = Visible Pixel
Pixel with 0 Alpha + Pixel with >0 Alpha  = Transparent Pixel
Pixel with 0 Alpha + Pixel with 0 Alpha = Transparent Pixel

This might explain it better if you're a visual learner:

It sounds like you want to modulate the mask alphas, so the ui is only visible where all of the masks alphas are 1.0 (with soft blending on values <1.0)?

It's probably easier and less error prone to do this in a single shader while rendering the ui component, but if the goal is to separate the layers into rendering passes, then something like this might help:


The code blends each mask layer on the frame alpha channel, then blends the ui component with the combined alpha afterward.


To control whether ui components are able to draw outside of their region you could also use the scissor test to limit rendering.

how would I do this in a single shader?

Also, I can't seem to get it to work.
I put the mask drawing under the // render masks where alpha 1.0 is visible and 0.0 is not... 
And I put the drawing under the // render ui control..

Renderer inherits from spritebatch. It just adds a white pixel, and a shortcut for drawing that white pixel: draw(rect,color) = draw(pixel,rect,color)


I'm not sure how this works, will this be able to combine masks like this?


If so, would you be so kind to post a basic example that includes all the drawing code?

Thank you for your time.

I thought I found a way, but it doesn't work on the computer...

Stencil Buffer:
0 0 0
0 0 0


First, I draw a black layer over the screen on the stencil buffer:
function: always
pass: replace

1 1 1
1 1 1

Then, I draw my mask
function: equal
pass: keep
fail: decrement

mask:
1 1 0
1 1 0

original:
1 1 1 > 1 1 0 > p p f > keep keep -1 = 1 1 0
1 1 1 > 1 1 0 > p p f > keep keep -1 = 1 1 0

Then I can draw any more masks to 'combine it' with the parent masks

1 1 0 > 0 1 1 > f p f > -1 keep -1 = 0 1 -1
1 1 0 > 0 1 1 > f p f > -1 keep -1 = 0 1 -1

Then, when there are no more child masks, I can revert the changes by drawing a black layer.
function: equal
pass: keep
fail: increment

Reverting
0 1 1
0 1 1

0 1 -1 > 1 1 1 > p f p > +1 keep +1 = 1 1 0
0 1 -1 > 1 1 1 > p f p > +1 keep +1 = 1 1 0

Then I revert my first mask:

Reverting
1 1 0
1 1 0

1 1 0 > 1 1 1 > f f p > keep keep +1 = 1 1 1
1 1 0 > 1 1 1 > f f p > keep keep +1 = 1 1 1

This SHOULD work, but for some reason it doesn't.  It results in the first black layer being drawn and the first black layer's failed parts aren't erased

IE:

I drew
1 1 1 1 1 > 0 0 1 0 0 (should be >) F F p F f > 0 0 1 0 0
1 1 1 1 1 > 0 1 1 1 0 (should be >) F p p F f > 0 1 1 1 0
1 1 1 1 1 > 1 1 1 1 1 (should be >) p p p p p > 1 1 1 1 1
                             (results in >) P P P P P > 1 1 1 1 1
                             (results in >) P P P P P > 1 1 1 1 1
                             (results in >) P P P P P > 1 1 1 1 1

Why does this not work???

cable729:

Renderer inherits from spritebatch. It just adds a white pixel, and a shortcut for drawing that white pixel: draw(rect,color) = draw(pixel,rect,color)

SpriteBatch changes the blend states.  For this to work with SpriteBatch you'll need to render the batch using Immediate mode, then apply the state changes after calling Begin and before calling Draw.

Also make sure the mask quads you're laying down all render to the same screen area, and use textures with alpha channels to apply the mask - this is important for the above code to work, it requires rendering the layer's invisible areas in order to clip previously rendered masks and help build up the composite mask.

Thanks! It works! Now all I need is a way to remove the mask i just drew

as in I drew

1 1 0
1 1 0
1 1 0

Then

0 1 1
0 1 1
0 1 1

=

0 1 0
0 1 0
0 1 0

Now I want to take back the mask:
0 1 1
0 1 1
0 1 1

To get
1 1 0
1 1 0
1 1 0

Thanks again!


EDIT:

Either that, or is there a better way to do this so I don't have to revoke the masks:

1. Apply Mask
2. Draw Self
3. Draw Children (go back to 1)

EX:

Panel A
Apply Mask
0 1 1
0 1 1
0 1 1
Draw Self
Draw Children
{
    Panel B
    Apply Mask
    1 1 1
    1 1 1
    0 0 0
    =
    0 1 1
    0 1 1
    0 0 0
    Draw Self
    No Children
    Revoke Mask
    =
    0 1 1
    0 1 1
    0 1 1

    Panel C
    Apply Mask
    0 0 0
    1 1 1
    1 1 1
    = (If panel B DIDN'T revoke it's mask)
    0 0 0
    0 1 1
    0 0 0
    = (if panel B DID revoke it's mask)
    0 0 0
    0 1 1
    0 1 1
   
    ETC.
}