Eye replacemant - First Test

Since I'm doing post-production on pretty much every final project that involves CG I get to really concentrate on the stuff I really want to do this year. There are some cool projects this year, but probably the most challenging task is an real-life eye replacement on a stop motion character. It's Sarah's idea and she took it from the Canadian stop motion short "Madame Tutli-Putli" which first involved life action eye replacement. On the web side it says that production for that 17 min short took 4 years in total. The main reason is that they had the actors act out the animated scenes and then masked out the eyes and compted them on top of the stop motion character. My first thought was: "Why don't they used 3D camera mapping?" Well as it turns out the compositor working on it is mainly a portrait painter and probably (I only assume) doesn't know 3D that well. So I came up with the idea of 3D motion track the movement of the stop motion character, then camera project the life action footage (taken from the front) onto geometry, which then is animated according to the 3D motion track.
Now to the testing!
Orla did a quick stop motion test. Because of technical difficulties, we had to shoot it in low res (788 x 576), while the life action footage is shot in HD 1080.
Here's the stop motion test!



Here's the life action front shot!



So... I tracked the stop motion test, but unfortunately I (again) messed up the set up. There were not enough tracking points to track all of the movement, without any jerky glitches. The first 2, 5 seconds are pretty much useless. Well I guess that's why we do the testing. The last 6 seconds were fine though, so at least I had something to play with.
I prepared (stabilized and masked) each eye in After Effects and exported them as 32bit EXRs.

Then modeled the geometry according to the 3D location of the locators exported from the 3D motion track. Because MatchMover allows you to animate the camera rather than the scene, it's possible that the geometry can just sit in 3D space, while the camera moves around it, making it look like the geometry is moving.
This is the geometry for the projection (not perfect. Gotta spend more time on it once it comes to production).


Next thing I build the set up for the camera projection and painted transparency onto the mesh using Maya 3D paint tool! Here's a screen shot of the geometry with the life action footage projected onto it!

The main thing here is that the camera is already tilted, but because the 2d(!) footage is projected onto 3D geometry it still looks 3D.

Maya's viewport allows to see the camera projected geometry displayed and updated, which makes it easier to set up the mesh and the cameras.

The shading and lighting is going to be a bit iffy. But I tried to simulate the lighting.
That's all the work done in Maya. Now it's comping!
I rendered out each eye and one render of the mesh with a lambert material override to get some shading. The comping mainly involved color grading, masking out the nose and adding some grain to the CG shots, since the resolution was much higher (although I think I got a bit carried away with that).
I also had to paint out some bits of the puppets eyebrows, that woulg stick out at the beginning when it turns around.

There're still some things to sort out, but the main idea stands. The test was mainly to go through the pipeline and to proof to Sarah that it could be done. She's not that familiar with 3D so she didn't trust the idea at start. But I think I managed to convince her that this technique is feasible and will save her a lot of time over the masking technique from Madame Tutli-Putli.