MMD Augmented Reality AR on Android Devices

VOCALOIDS IN AUGMENTED REALITY ON ANDROID DEVICES
AND OTHER COOL STUFF

The MMD program has been around more or less even since Miku Hatsune first made her debut on the world stage in 2007. The current version of MMD (version 9.26) has been around for nearly two years now and it doesn’t seem likely that it will be updated to take full advantage of current technologies.

So the question that begs to be asked is: “what’s going to happen next?”

Vocaloid technology is making huge advances; with each new engine, the voice quality is increasingly ‘human-like’. All you have to do is just listen to some of the songs sung by IA or recited by her sister, ONE.

In the Japanese MMD community, there is a lot of experimentation with migrating the vocaloid models to some of the new technology platforms now available. Technologies like VR and AR, and the devices on which these technologies run.

Having bought a tablet a couple of months back, I found it so useful that I am increasingly using far more often than my laptop or netbook. Which was why I started to look at what it can be used for in regards to MMD.

What I learned was that there are some really interesting possibilities and in this article we will examine just one method to use a tablet or smartphone for MMD in a creative manner.

WHAT IS POSSIBLE NOW?

Honestly, unless you have superb programming skills, not a lot. Plus both tablets and smartphones, being touchscreen devices with comparatively puny specifications as compared to contemporary laptops, don’t currently have the facilities to be able to do what MMD does so well.

Specifically: posing models, developing motions and rendering videos.

These limitations have not, however, stopped people from trying out new ideas that do work on tablets and smartphones. If you go to the Google Play Store, there’s actually quite a few MMD related apps available. Many are really just “proofs of concepts”, but interestingly, in terms of usable creative apps, the one most promising is an Augmented Reality app.

VOCALOIDS AND AUGMENTED REALITY

Using an AR app with vocaloid models is really an exciting concept. It allows you to take, for example, Miku Hatsune from out of her virtual world and into ours.

So, for example, instead of her being stuck around her old Gekido City haunts, she can now cause mischief in your neighborhood. Since this is now possible, even with the somewhat primitive AR apps available, users can now enjoy and easily share this new level of engagement through the many social network media with which tablets and smartphones are so easily connected.

The creative possibilities, should not be under estimated as literally these are almost limitless. Plus, it’s easier to do than using MMD!

To prove this point, this entire article, including the images used to illustrate it, are all done using a tablet. In case you’re thinking that my Android tablet has mad specifications, it doesn’t. It is a generic, basement price model that I picked up for under $100 at Best Buy.

APPS AND SPEC REQUIREMENTS

Hardware Requirements

All you need is a decent Android tablet or smartphone with decent graphic capabilities and sufficient memory. My tablet doesn’t have the greatest video card; it cannot run high demand graphic games such as Fire Emblem V, but I did buy

a 32GB SD memory card for it, so it has plenty of storage. It comes with two cameras (front/back) of somewhat lackluster capabilities. These can be improved to a point by using an aftermarket camera app, but there is a better work around. This will be covered below. However, it natively does have a screen resolution of 1280×800 pixels, the same as my old VAIO laptop. Also, it does by default, produce HD quality PNG format images. So overall, it does a pretty decent job in terms of picture making functions.

Apps Required

You will need the following apps:

An AR app that uses vocaloid models
An image editing program (optional)

The AR app I recommend is: [MIKU AR Camera] Mikuture

HOW AR APPS WORK

The concept is very simple. What an AR app does is to project an image of a vocaloid model over an image viewed through your device’s camera. In actual fact, although the app is dynamic in the fact that it can do this “live”, you could as easily use a still image for the background.

Honestly, you can do something similar using just MMD or MMD with Photoshop or GIMP. However, it is a lot easier using a VR app on a tablet or smartphone. In fact, it is ridiculously easy.

The current version of the app only has a limited number of model choices, also you’re stuck with having only about 30 poses to work with; fortunately, the poses can be slightly modified plus you have almost full access to the facials as offered as standard on the original MMD version of the model.

THE WORKFLOW

Making an AR picture couldn’t be easier; but I will outline my method below as it gives the best results. Since the Mikuture app is relatively straightforward, I will not discuss in detail how to use it. But I do recommend that you familiarize yourself with its features so that you can get the most out of it. It is seriously powerful.

STEP 1

Choose the model that you wish to use and set up your shot. Remember to lock the camera view once you’ve got the setup that you want.

STEP 2

Choose your pose and then fit the model properly into the scene. Perspective is important as is scaling, if you want to make your model life-sized. The model alignment controls are the same as those found in MMD.

STEP 3

Modify the pose! This will break up the stiffness. The head and eyes are the only parts of the pose you can really change, but the fact that you can is something that can be leveraged to advantage. There are also some hand poses which you can add; use these as well if applicable.

Make good use of the facial controls! These work the same as in MMD, and are instrumental in making the pose “yours”.

STEP 4

Use the lighting controls to blend the model into the scene. I cannot emphasize enough how important this step is if you want really convincing end results.

Once you are happy with the results, capture the image with the app’s control. This will save the image to your device’s storage. Note that you cannot save your work, so once you exit the app, it is all gone. This can also happen when you try to switch apps!

On my tablet the images are saved with .JPG extensions by the app. However, since the app uses my tablet’s internal screen capture functions to create the image, the images are actually .PNG files. So, I found that I need to manually correct these images by renaming them with the proper extensions.

STEP 5

Essentially, the picture is done and in general the results are quite pleasing. However, if you have a good image editing app such as either Pixlr from Autodesk or Photoshop Express from Adobe, it is well worth running the image through one of these apps for even better results. The image of Luka Megurine at the top of this article was enhanced using one of these apps.

STEP 6

Share your image via one of the many social network outlets available. With Android devices, this is really easy to do!

OTHER ANDROID VOCALOID APPS

If making images isn’t your “thing”, there are actually a lot of other types of apps that uses MMD models or anime style imagery of Miku or other vocaloid characters.

One of my favorites is the Miku 2D Anime LiveWallpaper shown below. This is an interactive app; Miku does react to what you do on the screen. She speaks (in Japanese), complaining if you poke at her too much. She even snoozes if you are not working on your device. Plus the app has a clock and battery level indicator built in.

There are also a lot of simple games that feature Miku or her friends, just use the search terms below in the Google Play Store app to find all the applicable apps. Most of them are quite good; just avoid for the moment the ones you have to pay for, as these are generally not worth spending money on.