Based on the searches I've been doing, it seems like this subject crops up every now and again. :-)

I'm looking at importing morph targets from a 2009 FBX file (generated by 3ds MAX). After a lot of searching and reading and general researchy stuff, probably the most valuble (and concise) item of information I've found is this sentence fragment from jwatte:

... or you can export as a morph target, and bind multiple vertex buffers to multiple streams, and blend between them using a vertex shader.

I've done some testing on a simple morph test model (consisting of a box with two animated morphs), and have skimmed through the fbx file itself, so I'm not flying completely blind here.

So here's my questions, as well as any info I have found. Any advice or suggestions, as well as any corrections in places where I'm wrong, are appreciated:

 ~ When importing, will the default importer bring in the different morph meshes in a format I can use? 
        - Initial testing indicates that it will not. The imported model structure of the test model consists of a single mesh, not three. This indicates that I will have to write my own importer.
        - Reading the FBX file indicates that the two morphes are defined by the syntax "Shape: "MORPHNAME" {" followed by a list of indices, followed by a list of vertex positions, followed by a list of normals. Are there any specific importer commands I should know about, or is this fairly standard material for the content pipeline?
 ~ When importing, will the default importer bring in the animation keyframes?
        - I already know I'll have to build a content processor to handle them, ala the skinning sample, but will the importer bring them in?
        - I can't actually understand what many of the values in the FBX file mean here, but I'm hoping that it won matter and the default importer will bring them in. If I'm wrong: any information about how this data is encoded is appreciated.
 ~ Will this be compatible with the skinning sample?
        - Rather than building this from scratch, I figured I would extend the skinning sample (because when all else is said and done, we need both skinning and morphing for character animation in this project). The main issue I can see is making the skinning sample shader take multiple vertex streams (something else I've never done before). Does anyone have any advice?


When I'm finished I'll probably document the code and make it available like I did with my QuadTerrain project. It seems like a good extension of the skinning samples functionality, especially since it is one of the default export options of the Max FBX exporter.

Thanks!
Qu.

I've spent a bit of time looking into my options, so far my thoughts are that making an FBX importer from scratch is probably more than I want to get myself into. I have been considering another option: a program that rewrites the FBX file before I import it, converting shapes into meshes, which are then imported with the default importer. I have done something similar (though much less complex) in the past: I built a small program that takes several animation clips in seperate files and combines them into several animation takes in a single file.

If I do this, I'm assuming that I'll be able to import the animation keys via a method similar to the method employed by the skinning sample.

I'd really appreciate some feedback on this, otherwise I suspect I'll end up wasting a lot of time. Anyone?

On the edge of OT, a bit of information about facial animation and morph targets.
Autodesk had a nice presentation on gdc09, you can find it here: http://area.autodesk.com/index.php/events/gdc_2009_videos, under the name "Ubisoft, Prince of Persia". Hope it's useful.

The XNA Model content format does not support morph targets. The X file format does not support morph targets. The XNA importer for FBX and X geometry does not support morph targets. Thus, anything you do will have to be custom at some level.

The easiest thing, if the number of morphs is small, is likely to build each morph target as a separate mesh (possibly instances of the same base mesh, to get the topology exactly right), and make them siblings in the node hierarchy. After you load the Model, you have to walk through the ModelMesh parts, and bind each of the vertex buffers to a separate vertex stream, and set a vertex declaration that taks position/texture/normal/whatever from stream 0, and only takes position1/position2/position3 from the other vertex buffers. Then bind an Effect with a vertex shader that knows how to morph between the different position streams. If you want to combine this with skinning, you can still use weight/index data in position 0, and a matrix palette, in the same effect -- but you will have to write the effect custom; the Skinning Sample won't work.

The XNA animation importer only knows how to import bone translation, rotation and scale animations (keyframed transform matrices). This means that other animation parameters (light brightness, camera FOV, morph weight, or what have you) is not imported by the stock XNA content model. To import and apply such animations, you will have to write your own animation importer, animation content format, and animation classes to sample the animated value.

My advice if you want to use morphing: Give up XNA, go buy Granny3D, and develop in native code :-)

jwatte:

My advice if you want to use morphing: Give up XNA, go buy Granny3D, and develop in native code :-)

Okay, now I'm scared. :P
But still determined. We shall not surrender! Never! (Insert 300 quote here)

Big thanks for responding with so much detail, jwatte!


 

I want to avoid writing a custom importer for FBX, so here's my approach angle so far:

1) Make two copies of the same thing: one with three morph targets (FBX A), the other with three meshes (FBX B).
2) Build a seperate program to textually convert FBX A to FBX B. (This allows me to export morph targets, run them through this converter, and import them as meshes)
3) See whether it works for other morph target files, and make adjustments until it works properly.
4) Build another seperate program (or a different module of the same program) to remove the morph weight animation data and build it into a seperate file, which I import into XNA with a custom importer.
5) What you said, about using several vertex streams, and messing about with the Effect to combine them based on the custom animation data... I'll need to do a bit more research on the subject, but I should be able to manage it eventually. Hopefully. :)

The final usage process: Export from Max >> Run through external programs >> Import model file and animation file >> Make a custom MorphAnimationPlayer >> Link it to the Model and MorphAnimationData >> Feed all this to the Effect, which will do the morph merging.

This is going to be fun! Quick question though: will using a shader that links to several vertex streams use up more than a couple constant registers? I'm wondering how this is going to affect my bone limit (which is 80, in my current skinning system)...

Vertex input streams are independent of constant registers, so this won't affect your constant usage.

Shawn Hargreaves:

Vertex input streams are independent of constant registers, so this won't affect your constant usage.

"Excellent..."

Thanks for that Shawn! :)

I will see what I can see!

I have spent some time familiarising myself with the fbx format, and found some of this stuff quite interesting, so I thought I'd share. You already know that I'm building an app to convert morphs to meshes: here's the details:

Morphs are stored as three comma-delimited lists.
 ~ The first is indices: vertices that are not changed by the morph are not indexed, and no data is stored regarding them. This presents a problem: either I store these indices somehow, or I add 'null vertices' and increase the size of the vertex buffer for morph meshs which don't change much. I haven't yet decided, and although I'd prefer to do the latter, the former would be easier.
 ~ The second is vertex positions. These are stored as delta's: the vector from the original position to the morphed position. At first I thought that I would have to recreate the original positions for my morphed meshes: I have since decided that it would be easier and more efficient to just store the delta's as the vertex positions.
 ~ The final is normal direction, also stored as delta's from original.

Morph animation is stored seperately
 ~ Which is ideal for my purposes: I simply remove the animation section from the FBX file entirely, and copy it into a custom file format (*.mta, for Morph Transform Animation).
 ~ I already have worked out that the first two values are Time and Weight, and that all the keys for a single morph are stored in the same list (no delimiters), but if I want to do anything more complex than linear interpolation, I'm going to need to work out what the other values stand for. Does anyone know of any good resources on this? Perhaps Autodesk provides the FBX format details somewhere...

Qu:

Perhaps Autodesk provides the FBX format details somewhere...

The FBX file format is not documented (and tends to change with each version of the FBX SDK) but Autodesk provide a free C++ API and object model for working with these files. I would recommend using that rather than trying to parse the disk file directly yourself.

Shawn Hargreaves:

The FBX file format is not documented (and tends to change with each version of the FBX SDK) but Autodesk provide a free C++ API and object model for working with these files. I would recommend using that rather than trying to parse the disk file directly yourself.

Oh... [puts on dunce hat]... whoops... I'm already mostway though the extraction function and am writing the data rearrarangment function...

I'm not sure: an API would definately be easier to use, and would make it compatible with future versions, but I'm already halfway done with this, I've never used C++ and I'd need to download C++ Express... [scratches head]

I'm downloading the API, but I may simply end up keeping what I've already done and either updating it to suit each new FBX format, or simply get the artist to export to 2009 format. \:)

Qu, wouldn't it be a lot easier simply to export a custom format from your 3d modeling app, and then load the data in your game?

Daaark:

Qu, wouldn't it be a lot easier simply to export a custom format from your 3d modeling app, and then load the data in your game?

Correct me if I'm wrong, but wouldn't this require writing a complete exporter and a complete importer for my models?

I'm trying to chain this thing onto the default FBX exporter and importer, so that I can keep using all of the stuff that already works fine. The FBX exporter can export morph objects and the importer can handle mesh objects, and the two object types are quite similar, so it seems like writing a converter is the most harmonious solution.

Qu:

Correct me if I'm wrong, but wouldn't this require writing a complete exporter and a complete importer for my models?

Of course, but how hard is that?

Just come up with a class that holds all your info, and then read and write that into a file. You can skip the content pipeline stuff, and just write a simple function to read from an included file. There are many harder things to do than reading a bunch of floats from a file. Most modeling apps have a built in scripting language, and you can easily just output all the values in whatever file you want. Then you just read that into a custom morph-able model type.

Daaark:

Qu:

Correct me if I'm wrong, but wouldn't this require writing a complete exporter and a complete importer for my models?

Of course, but how hard is that?

Just come up with a class that holds all your info, and then read and write that into a file. You can skip the content pipeline stuff, and just write a simple function to read from an included file. There are many harder things to do than reading a bunch of floats from a file. Most modeling apps have a built in scripting language, and you can easily just output all the values in whatever file you want. Then you just read that into a custom morph-able model type.

Well, I'm naturally wary of messing with things I've no experience in (MAX Scripting, for instance :P), but I guess my whole point is that the standard pipeline already does all this for me. Why reinvent the wheel, when I can simply make a few modifications and use the pre-existing model class and pipeline? Your way just seems like 4x the work for the same end result...

Qu:

Well, I'm naturally wary of messing with things I've no experience in (MAX Scripting, for instance :P), but I guess my whole point is that the standard pipeline already does all this for me. Why reinvent the wheel, when I can simply make a few modifications and use the pre-existing model class and pipeline? Your way just seems like 4x the work for the same end result...

My way was trying to save you work instead of jury-rigging a bunch of existing things with shoehorned features. You are having a hard time, and you are having to mess with FBX files. You don't need to become an expert at Max script. You only need to find out how to write out the data from your model's morph targets (or whatever Max calls them).

You're afraid to use both maxscript and C++ to make your job easier, and you're telling me I'm making it 4x harder by offering an easier solution? :)

Ah right, I think I'm following you now. It sounded like you were suggesting exporting the complete model and all accompanying data (materials, bones, animation and of course morphs) to a custom format, rather than simply the morph targets. Hence my 4x estimate. :)

I still don't know that it's an easier solution, though. Messing with FBX files textually isn't difficult: they're readable and logical, and it's convenient: the morph meshes become part of the same Model object. More importantly: the morphs are already exported as a list of floats within the FBX file: it's just as easy to extract these as it would be to extract them from a custom format. And since the format I need them in within XNA is a vertex buffer, converting to a mesh and getting them imported that way seems like the logical choise.


The most difficult part is building a new mesh from the morph data: I keep going backwards and forwards between null vertices (which would make the entire project easy to program, at the expense of performance and memory) and storing indices in UV coordinates (which requires a much more thorough system, to change the size of the meshes vertex buffer and check each index against it's correct vertex). I've been spending my time trying to get the latter to work...

Apart from this one sticking point, though, I'm not having a hard time with it at all. I've been able to import a mesh with it's morphs removed by the converter, so if I can get the actual morphs themselves working I'm home and dry.

Thanks for your suggestions though, they are appreciated! :D

I'd like to re-iterate my previous suggestion for how to most easily do this. I'm assuming you're using 3ds Max; Maya or others should be similar.

1) Build the base mesh, name it "somename:base"
2) Reference the base mesh, call the reference "somename:morphA" and then apply an Edit Mesh or Morpher modifier on the reference
3) Reference the base mesh again, call the reference "somename:morphB" and then apply a different Edit Mesh or Morpher modifier on the reference.
4) Export as FBX (or as .X)
5) Import as normal

Now, you have two options.
A: Read the vertex buffer of each of the meshes at start-up. Calculate and store the position deltas, as well as the base mesh data. To draw a morphed version, add the morph deltas into a copy of the original base mesh data, and then draw the base mesh. This requires CPU work at runtime.
B: Write a content processor. This content processor derives from ModelProcessor, and finds meshes named "whatever:base" and treat them as base mesh, then finds meshes named "whatever:somethingelse" and treat them as morphs. Extract each of the position channels for the alternate meshes, and add as POSITION1, POSITION2, ... channels in the original mesh vertex buffer. Delete the morph meshes. Store a tag that maps morph names to POSITION channel indices. At runtime, send in vertex constant data for how much to use each channel -- for example, if you have POSITION0 ... POSITION4, you could use a single vertex register, and calculate position as:

struct VSIn {
  float4 position0;
  float3 position1;
  float3 position2;
  float3 position3;
  float3 position4;
}
float4 weight;
VSOut Transform(VSIn in) {
  float4 vPos = in.position0 + float4(weight.x * in.position1 + weight.y * in.position2 + weight.z * in.position3 + weight.w * in.position4, 0);
  xPos = mul(vPos, WorldViewProjection);
  ...
}

Just make sure that the topology of the meshes is the same -- don't change material assignment, smoothing groups or faces in the versions of the mesh that define the morph targets.

In my opinion, this is both the easiest way to get morph targets into XNA, AND the easiest way to author morphed data in the absence of proper morpher support in the content pipeline.

Jwatte: thanks for your post! I already had a general idea of what I was going to do, but your post went into enough specifics to give me a really good idea of where I'm headed!

In essense, exporting as seperate meshes is precisely what I'm already trying to do, except that I make the seperate instances from the morph data iself during the whole 'conversion' thing, rather than making the artist do it manually. Combined with extracting the animation data from the same file, it's really easy to use on their end: very nearly a WYSIWYG animation system. It's more work for me, but less for them, and when I try to explain it to them it makes them think I'm an epic programmer. ;-) Plus it means I can skip the "calculate the delta values" step.

I really like your content processor idea: I was going to use vertex streams, indexing into a seperate Buffer for each morph-weight value, but adding all the morph data to a single meshes vertex format sounds a whole lot easier. What happens if we have >4 morphs, though? I'm still a newbie when it comes to vertex streams, but I'm assuming that I would create a second one with another 4 position values, and then another if we've got >8, etc?

I'm enjoying this! Thanks for the continued help guys! :)

There is a limit to the number of vertex streams that a vertex shader can read at the same time. There may also be a limit to the number of indices you can define per semantic. I don't know what the limit is -- it might be as low as, say, 16. (The D3DVERTEXDECLARATION9 struct runs out at 63 + termination element). If you have a large number of morphs, you pretty much need to do it on the CPU, unless you have floating point render targets and render target type casting ("render to vertex buffer") -- which you don't in XNA.

Thanks for your help! I'm still working my way through it: haven't reached runtime code yet, but the convertor is finished. It takes Morphs and turns them into what I've been calling 'delta meshes,' where the morph deltas are stored as positions, as well as extracting the animation data as a series of keyframes and storing it in a second file.

I've started writing the content processor, and have run into a problem mentioned here:

CharlesXna:

Just a warning from someone who implemented this:  the normal model processor sorts vertices for efficient application on the graphics cards.  So, if you load multiple meshes don't count on the vertices corresponding.  I had to write my own custom importer to get morph targets to work that really is doing the same thing as the normal importer with an extra bit of code that will it to not so that sort on morph targets and morphing meshes.

Despite the test file having 26 vertices in both meshes (it's a cube, split once along each side), by the time it reaches a NodeContent object I've got 54 vertices in my base mesh and 48 in my morph mesh. (I think) I'm trying to work out a solution, but it's slow going.

Note that vertices will be split based on material, UV and normal discontinuities. That's why you have to use the exact same materials/UVs/smoothing groups/topology in the morphs as you do in the original mesh -- I think I mentioned that above, but perhaps not the reason why.

One trick is to export the mesh with a separate vertex channel for "vertex number," which is simply a channel that numbers the vertices based on the source mesh. You can then use this to see which split verts belong to which original verts. You can, for example, enter this into an unused UV channel.
(Hmm... would be cool if kW X-port could do that for you... must add that to "the list")

jwatte:

Note that vertices will be split based on material, UV and normal discontinuities. That's why you have to use the exact same materials/UVs/smoothing groups/topology in the morphs as you do in the original mesh -- I think I mentioned that above, but perhaps not the reason why.

Hmm, that gives me an idea. After conversion, the morph meshes are 90% a copy/paste of the original mesh, with the extra 10% being names, positions and normals. So far, I've never seen anything at all stored in morph normals: they end up as a list of zero's no matter what I export. If I was to copy the original meshes normals into each morph mesh, then I might just get away with it: each vertex mesh would be split identically. Maybe.

Easy enough to test, and if it doesn't work I'll try setting up a seperate index channel system... maybe I can use the normals for that too? :)

Progress, made, it has been. [/Yoda]

I have sucessfully imported a morph! The content processor was fairly easy to make, and after that it was almost entirely a HLSL matter. Not that there weren't a few hiccups along the way, including discovering that the importer simply can't handle a massive string of comma-delimited vertices without a single return, but it's all good now: I can use O and P to morph between the base mesh (a teapot) and the morphed version (a spherified teapot: I call it the BubblePot^TM! :D ).

Of course, it's not scalable at all at the moment:
 ~ The number of morphs (1) is hardcoded in the shader (a problem I'm not sure how to fix: I guess I just make a seperate shader for each character with a different number of morphs?),
 ~ The system is standalone at the moment (not combined with skinning)
 ~ I've not tested it with multiple meshes (although by all accounts it should work).
 ~ I'm also wondering how I can get morphed normals through: I don't think they're even exported in the first place! But I'll be rather disappointed if the lighting on their face suffers because I can't generate morphed normals... any ideas?

PS: Something just occured to me: this system should be perfectly compatible with exporting the morphs manually, rather than using the convertor! If the user names the morphed mesh: "[Basename]_[Morphname]_MT" then the export result should be almost identical to the same thing made with the convertor! I'll have to test this theory out... :)

Okay, I've been working on upgrading both the extractor and importer for the raw animation data. This turned out to be a whole heap more trouble than I expected: the FBX format isn't all that logical when it comes to storing keyframes...

It stores it in a solid block of comma delimeted values, with no marker between the end of an old keyframe and the start of a new. Each keyframe appears to have either 9 values if you export as automatic curves, or 7 values if you add special curve properties (in which case, it exports a key-frame for every frame).

My artist has infomed me that for the most part, he'll be using frame-by-frame animation for facial expressions. This makes things nice and simple: all I need do is import the data and interpolate between them (probably as a Catmull-Rom curve)

After extracting the few nuggets of goodness from this bloated flub of weird data to my new MTA animation file, which consists entirely of:


I went to work on an importer for the file. To cut a long and unexpectedly painful story short, that is done: I now have the data in a MorphModelAnimationData object, which has a list of MorphMeshAnimationData objects, each of which has a list of MorphAnimationChannel objects, each of which has a list of MorphKeyframe objects.

All that remains is decoding this data into a list of morph weights, which I can hand to my wonderful shader as an array of floats.

The number of morphs is still hardcoded: I see no way around this at present, and suspect I will just have to make a seperate shader for each character with a different number of morphs. I'm not too concerned.

I am a little worried about the normal thing, though: morphing still only affects position, not normals. Our character model will eventually be normal-mapped, and I have absolutely no idea what effect this will have on lighting. As always, any suggestions, criticism or help is appreciated: if not, I guess I'll just keep doing what I've been doing and get the darn thing functional, over the weekend or early next week at the latest.

Cheers,
Quasar.

I'm curious as to if you got this working or not?  We have morph targets and full lipsync animation working in XNA in our game engine.  I'll try to get around to making a standalone demonstration sometime soon.  I would like to have it for my class, anyway.  I have custom content processing, but not a custom importer.  I import from FBX.  I use mesh names to tell the processor what to do.  A morph target is named something like Morph Target Face.  Then a morphing mesh is named something like Morph Face Smile.  We do the actual vertex morph (vertices and normals) in XNA, not in the shader. We have custom importers for the Autodesk xaf format since FBX does not export morph animations and we have a custom importer for Magpie files. I've seen the demos where they use the shader to morph, but that only works if you have a small number of morph meshes.  We have about a dozen for lip sync and expressions.  We've had no problem with performance with usually 2 - 4 characters on screen.  It is very sensitive to anything you may do to the mesh in 3DS Max.  Doing a Reset XForm will cause all kinds of problems. 

We've had this theory for a while that we're the only one's to get morphing actually working in XNA (about 750 lines of lip-synced dialog no less).  I'd love to hear from anyone else who's done it and compare notes.

Charles