Create MMD Lip-Sync Animation as a Separate File

How do I do lip-syncing animation to match my music in MMD? Is there an easy way to lipsync in MikuMikuDance? How can I match my model’s mouth movements to my music WAV file?

Lip-Sync Animation is Detailed Work!

I wish there were some secrets to making it easy … but I haven’t found any yet … it is simply tedious work to be enjoyed, if you can, until you have matched your soundtrack from end to end.

The basic rule is to use only two or three sliders at a time. I set the grin at about a 25% value and then leave it alone for now. Then, using the “a” the most often (to control the open mouth) … and the “I” (to control the width of the mouth) … and adding some “u” or “o” to get those sounds) you are off and running. Almost all of those other possible mouse settings can be used for facial effects … but they don’t help you to “see” the pronunciation of the words.

Just start at the beginning. SAY the first phrase to yourself and feel your mouth change shape as you say each word … I … won’t … grow … up. Exaggerate the motion of your lips as you say those words. Imagine you are in front of a large audience with no microphone. You want to speak loudly and clearly … how does your mouth move?  First, your mouth is calm, maybe a little open as you breath and smile … Now “feel” your lips and your mouth as you slowly, in an exaggerated way, say: EYEeee … WO..N.’T … G.R..OOO … Uh.P.

Have “Play WAV with Frame” turned on, keep an eye on your oscilloscope so that you can see the start of each word. NOTE that the Play WAV with Frame is not precisely accurate. Rely more on the oscilloscope for precise syllable placement. NOTE, too, that it is often best to start the animated mouth movement just a frame, or two, before the start of the visible wave … just works out better, for me.

As you work, you will want crisp movements. Feel how quickly your lips change shape as you read this sentence out loud. You don’t want MikuMikuDance to make smooth motions from one lip position into the next. Say “Lip Position” … it’s two words. You can feel the stop between them. You would want to animate a single frame of slightly open mouth between the two P sounds. To get the sharp movements, you have to register all of the mouse vowels you are using almost every time you need to change shapes. Like cutting camera scenes, you want to register the previous position right before you create the new position. The movements need to be sharp and crisp, not smoothly flowing as MMD is always trying to do.

There is only one real shortcut …

There is only one real shortcut in the process of creating your lip-sync animation: most songs have some repetition of the lyrics … so you can carefully lip-sync your way through that lyric the first time … and then copy/paste those diamonds into place every time that phrase comes around. “Oh baby baby I love you!”  or, in my example “I won’t grow up!” … You may have to do some fine tuning to get the pasted data to better-fit the soundtrack … but it’s a lot faster than creating a new sequence for each repetition.

Make your motions boldly …

Make your motions in a bold way. Open that mouth … flap those lips. A little too extreme is better than not enough action. At the bottom of the EDIT MENU is “Multiply of Facial Expression”. You can use that function to pump up your motions if you didn’t make them strong enough to begin with … and you can also tone down the motions if you feel you’ve made them too strong … better to start with them a bit strong so that we can see the lips flapping on that small mouth in the distance rather than so weak that we see nothing, at all. Stronger motions provide good data for the “Multiply of Facial Expression” tool.

You will find that different models require more or less “boldness” in your mouth motions. That’s another place where the “Multiply of Facial Expression” tool can help you … you can match the motions to your model’s requirements.

Do your Lip-Sync Animation in a separate dance file!

When you are working on a new project, it’s easier to do your lip-sync work with a model that is standing still!  If you have already begun your new dance and have your models in motion, the model’s gyrations may keep you from seeing the model’s mouth well enough for you to get the job done.

The first step in any project is to get your soundtrack finished. Add some silence for the first few moments … then maybe add some applause … start your music so that it overlaps the applause a little bit … add some silence in the middle if you need to … and maybe a bit of applause and a bit of silence at the end, as well.

With your soundtrack complete, you can start your MikuMikuDance project! You could do the mouthing of the words first … before any other motions are created. That way it will be done! … and you can start work on the dance, itself. But the lip-sync animation is so tedious that you won’t be able to wait until that’s finished before you start on the fun part … the actual animation. A 3-minute song can take five or six hours spread over several days of sliding those mouse (mouth) sliders to get what you want.

So … to get right into the animation while I am still excited about it, I can start my project and, then, make a separate dance-file just for the lip-flapping lip-sync animation. When that is finally complete, you can “select all facial” using the Range Select function and do a SAVE MOTION to create a VMD file … or you can copy/paste the motion into your main project file.

Good lip-sync animation adds so much to your finished video! See, for instance, the Departures memes on YouTube. That meme is all about perfect lip-sync: http://youtu.be/NrKqGwlWzAY

Enjoy!

– SEE BELOW for MORE MMD TUTORIALS…

— — —

– _ — –