Game Algorithms

Re: Facial Animation and Morph Targets

Stainless — Tue, 22 Sep 2009 08:50:17 GMT

Honestly I had it working on an iPaq, the trick is getting the textures from your art package in a painless way. After that it's trivial.

I was lucky, the application took a photo of a person (taken by a digital camara attached to the iPaq) emailed it to a company in the US that returned a 3D mesh in a standard format.

The 3D mesh was morphed to match the photo, so you got a 3d mesh of the target.

Since it was in a standard format, I could build the expression textures before hand and just apply them to the mesh.

It was very easy, but very impressive.

Re: Facial Animation and Morph Targets

Qu — Mon, 21 Sep 2009 23:02:31 GMT

Charles: That sounds like a smart way of doing things, particulary exporting the animations as XAF files. My system is a little different (okay, heck, it's a lot different) in that the morph meshes aren't actually recognisable meshes, they're a series of delta-positions (positions representing the distance each vertex moves when morphed). This, along with the animation data, is extracted directly from the FBX file during an intermediate step (between exporting from MAX and importing to XNA).

I love the idea of making the face a seperate rigid object attached to the head-bone: it makes way more sense than trying to build a combination skin-morph system (given that I'm morphing in the shader, it's more of an issue for me than you), and as you said keeps the # of morphed vertices down.

I'm kind of regretting the delta-position choise at the moment, because apparently exporting to FBX doesn't maintain normal-morphs, which may affect lighting (my vertex positions will be morphed, but my normals will not, and I don't know any way to re-generate them). I don't yet know if this will be hugely apparent or if I can afford to write it off.

Stainless: That's not a bad idea. A texture would be an excellent way to import large quantities of morph data without hurting the # of shader constants, and you could even mix them in XNA before bringing them into HLSL. Hmm... [mind working]

Gregoroamin: Well, I was going to say because I've got an artist who refuses point blank to use bones for facial animation, because morph targets are a whole world easier. But Stainless has a point: direct vertex manipuilation is a much faster option than using a load of different face bones.

Re: Facial Animation and Morph Targets

Stainless — Sun, 20 Sep 2009 09:32:40 GMT

Speed.

Doing all the morphing in a vertex shader is a lot faster than doing it on the cpu in c#, especially when you want to mix several morph targets.

Re: Facial Animation and Morph Targets

Gregoroamin — Sat, 19 Sep 2009 17:10:43 GMT

Why use totally unsupported morph targets when bones are supported? Its more work up front but a face can be rigged for every phoneme. Then do bone blending in code.

Re: Facial Animation and Morph Targets

Stainless — Wed, 16 Sep 2009 09:09:24 GMT

This is probably way too late but I thought I would stick my idea down here in case it is of use to anyone.

Store the morph targets as textures.

It would be fairly easy to use the fbx api to extract the morph targets from a file and store them as textures.

Then export the base mesh and import it into XNA as a model.

In HLSL you could take the base mesh and any number of textures and mix them in the vertex shader. So you could have 10% happy, 25% sad, 55% angry, 10% frown.

Just an idea, but I know it works as I did almost the same thing a few years ago to animate a talking 3d head using a text to speech system as the input.

Re: Facial Animation and Morph Targets

CharlesXna — Tue, 15 Sep 2009 17:00:02 GMT

We do the models and morphing meshes in 3DS Max and export in FBX as usual. Since the importer does not import the morph modifiers, we simply name the meshes appropriately so our content processor can detect them when we load. As an example, we might have a face mesh and four emotion meshes for smile, frown, anger, sad. The face mesh will be named: Morph Target Face in 3DS Max. Each of the morphing meshes will be, in turn named: Morph Face smile, Morph Face frown, Morph Face anger, Morph Face sad. We set up the normal Morph modifier in 3DS Max and create animations. We export the animations as XAF files. We have a custom importer and processor for XAF files that detects the animations for the morph modifier and renamed them so they apply to the correct avar.

The biggest thing we've learned in XNA is that you can pass a lot of information from 3DS Max or Maya to XNA in the naming convention you use. We use this to pass collision zones, camera zones, cameras, hotspots, doors, and all kinds of other things.

I have combined the morph with the skinned animation, be we don't in the any of the models in our game. Since the only things we morph are faces, we cut the face from the body mesh and just make it a child of the head bone. This keeps the number of morphed vertices relatively small. (We are also morphing another specific part of the male anatomy, but, that's a different story). We do the morphing on the processor side rather than the shader, so there's not really an issue to combine them.

Re: Facial Animation and Morph Targets

Qu — Wed, 09 Sep 2009 23:28:20 GMT

Hey Charles.

I haven't made any further progress on this, mostly because the system isn't vital until my artist actually comes up with some morphed models.

Your system sounds very impressive. I do the animation in the shader, and the limit is still a back-of-mind worry, but I haven't pressed any further with it so I figure I'll deal with it when I have to. How do you generate and export/import your animation? My system allows the artist to do this in 3ds MAX, which is handy.

I'm not exactly at a release stage yet, but I plan to get it there when I've got a pressing reason to work on it. I'd love to compare notes, too.

Also, have you combined it with skinned animation at all?

Re: Facial Animation and Morph Targets

CharlesXna — Tue, 01 Sep 2009 03:41:08 GMT

I'm curious as to if you got this working or not? We have morph targets and full lipsync animation working in XNA in our game engine. I'll try to get around to making a standalone demonstration sometime soon. I would like to have it for my class, anyway. I have custom content processing, but not a custom importer. I import from FBX. I use mesh names to tell the processor what to do. A morph target is named something like Morph Target Face. Then a morphing mesh is named something like Morph Face Smile. We do the actual vertex morph (vertices and normals) in XNA, not in the shader. We have custom importers for the Autodesk xaf format since FBX does not export morph animations and we have a custom importer for Magpie files. I've seen the demos where they use the shader to morph, but that only works if you have a small number of morph meshes. We have about a dozen for lip sync and expressions. We've had no problem with performance with usually 2 - 4 characters on screen. It is very sensitive to anything you may do to the mesh in 3DS Max. Doing a Reset XForm will cause all kinds of problems.

We've had this theory for a while that we're the only one's to get morphing actually working in XNA (about 750 lines of lip-synced dialog no less). I'd love to hear from anyone else who's done it and compare notes.

Charles

Re: Facial Animation and Morph Targets

Qu — Fri, 26 Jun 2009 02:27:30 GMT

Okay, I've been working on upgrading both the extractor and importer for the raw animation data. This turned out to be a whole heap more trouble than I expected: the FBX format isn't all that logical when it comes to storing keyframes...

It stores it in a solid block of comma delimeted values, with no marker between the end of an old keyframe and the start of a new. Each keyframe appears to have either 9 values if you export as automatic curves, or 7 values if you add special curve properties (in which case, it exports a key-frame for every frame).

My artist has infomed me that for the most part, he'll be using frame-by-frame animation for facial expressions. This makes things nice and simple: all I need do is import the data and interpolate between them (probably as a Catmull-Rom curve)

After extracting the few nuggets of goodness from this bloated flub of weird data to my new MTA animation file, which consists entirely of:

    
            ~meshname~   
        
                *morphname*   
        
                    :time, weight:   
        
                    :time, weight:   
        
                    :time, weight:   
        
                *morphname*   
        
                    :time, weight:   
        
                    :time, weight:   
        
                    etc...

I went to work on an importer for the file. To cut a long and unexpectedly painful story short, that is done: I now have the data in a MorphModelAnimationData object, which has a list of MorphMeshAnimationData objects, each of which has a list of MorphAnimationChannel objects, each of which has a list of MorphKeyframe objects.

All that remains is decoding this data into a list of morph weights, which I can hand to my wonderful shader as an array of floats.

The number of morphs is still hardcoded: I see no way around this at present, and suspect I will just have to make a seperate shader for each character with a different number of morphs. I'm not too concerned.

I am a little worried about the normal thing, though: morphing still only affects position, not normals. Our character model will eventually be normal-mapped, and I have absolutely no idea what effect this will have on lighting. As always, any suggestions, criticism or help is appreciated: if not, I guess I'll just keep doing what I've been doing and get the darn thing functional, over the weekend or early next week at the latest.

Cheers,
Quasar.

Re: Facial Animation and Morph Targets

Qu — Wed, 17 Jun 2009 21:51:59 GMT

Progress, made, it has been. [/Yoda]

I have sucessfully imported a morph! The content processor was fairly easy to make, and after that it was almost entirely a HLSL matter. Not that there weren't a few hiccups along the way, including discovering that the importer simply can't handle a massive string of comma-delimited vertices without a single return, but it's all good now: I can use O and P to morph between the base mesh (a teapot) and the morphed version (a spherified teapot: I call it the BubblePot^TM! :D ).

Of course, it's not scalable at all at the moment:
~ The number of morphs (1) is hardcoded in the shader (a problem I'm not sure how to fix: I guess I just make a seperate shader for each character with a different number of morphs?),
~ The system is standalone at the moment (not combined with skinning)
~ I've not tested it with multiple meshes (although by all accounts it should work).
~ I'm also wondering how I can get morphed normals through: I don't think they're even exported in the first place! But I'll be rather disappointed if the lighting on their face suffers because I can't generate morphed normals... any ideas?

PS: Something just occured to me: this system should be perfectly compatible with exporting the morphs manually, rather than using the convertor! If the user names the morphed mesh: "[Basename]_[Morphname]_MT" then the export result should be almost identical to the same thing made with the convertor! I'll have to test this theory out... :)

Re: Facial Animation and Morph Targets

Qu — Tue, 16 Jun 2009 00:16:44 GMT

jwatte:
Note that vertices will be split based on material, UV and normal discontinuities. That's why you have to use the exact same materials/UVs/smoothing groups/topology in the morphs as you do in the original mesh -- I think I mentioned that above, but perhaps not the reason why.

Hmm, that gives me an idea. After conversion, the morph meshes are 90% a copy/paste of the original mesh, with the extra 10% being names, positions and normals. So far, I've never seen anything at all stored in morph normals: they end up as a list of zero's no matter what I export. If I was to copy the original meshes normals into each morph mesh, then I might just get away with it: each vertex mesh would be split identically. Maybe.

Easy enough to test, and if it doesn't work I'll try setting up a seperate index channel system... maybe I can use the normals for that too? :)

Re: Facial Animation and Morph Targets

jwatte — Mon, 15 Jun 2009 23:24:55 GMT

Note that vertices will be split based on material, UV and normal discontinuities. That's why you have to use the exact same materials/UVs/smoothing groups/topology in the morphs as you do in the original mesh -- I think I mentioned that above, but perhaps not the reason why.

One trick is to export the mesh with a separate vertex channel for "vertex number," which is simply a channel that numbers the vertices based on the source mesh. You can then use this to see which split verts belong to which original verts. You can, for example, enter this into an unused UV channel.
(Hmm... would be cool if kW X-port could do that for you... must add that to "the list")

Re: Facial Animation and Morph Targets

Qu — Mon, 15 Jun 2009 22:19:15 GMT

Thanks for your help! I'm still working my way through it: haven't reached runtime code yet, but the convertor is finished. It takes Morphs and turns them into what I've been calling 'delta meshes,' where the morph deltas are stored as positions, as well as extracting the animation data as a series of keyframes and storing it in a second file.

I've started writing the content processor, and have run into a problem mentioned here:

CharlesXna:
Just a warning from someone who implemented this: the normal model processor sorts vertices for efficient application on the graphics cards. So, if you load multiple meshes don't count on the vertices corresponding. I had to write my own custom importer to get morph targets to work that really is doing the same thing as the normal importer with an extra bit of code that will it to not so that sort on morph targets and morphing meshes.

Despite the test file having 26 vertices in both meshes (it's a cube, split once along each side), by the time it reaches a NodeContent object I've got 54 vertices in my base mesh and 48 in my morph mesh. (I think) I'm trying to work out a solution, but it's slow going.

Re: Facial Animation and Morph Targets

jwatte — Thu, 11 Jun 2009 20:54:09 GMT

There is a limit to the number of vertex streams that a vertex shader can read at the same time. There may also be a limit to the number of indices you can define per semantic. I don't know what the limit is -- it might be as low as, say, 16. (The D3DVERTEXDECLARATION9 struct runs out at 63 + termination element). If you have a large number of morphs, you pretty much need to do it on the CPU, unless you have floating point render targets and render target type casting ("render to vertex buffer") -- which you don't in XNA.

Re: Facial Animation and Morph Targets

Qu — Thu, 11 Jun 2009 06:08:42 GMT

Jwatte: thanks for your post! I already had a general idea of what I was going to do, but your post went into enough specifics to give me a really good idea of where I'm headed!

In essense, exporting as seperate meshes is precisely what I'm already trying to do, except that I make the seperate instances from the morph data iself during the whole 'conversion' thing, rather than making the artist do it manually. Combined with extracting the animation data from the same file, it's really easy to use on their end: very nearly a WYSIWYG animation system. It's more work for me, but less for them, and when I try to explain it to them it makes them think I'm an epic programmer. ;-) Plus it means I can skip the "calculate the delta values" step.

I really like your content processor idea: I was going to use vertex streams, indexing into a seperate Buffer for each morph-weight value, but adding all the morph data to a single meshes vertex format sounds a whole lot easier. What happens if we have >4 morphs, though? I'm still a newbie when it comes to vertex streams, but I'm assuming that I would create a second one with another 4 position values, and then another if we've got >8, etc?

I'm enjoying this! Thanks for the continued help guys! :)