Let’s go straight to the point: here is a post about the use of WebGL for camera tracking. Usually the compositing is made with a software (After Effects, Motion, Nuke…) and rendered as a single movie file. Instead I’m gonna use Seriously.js by Brian Chirls. It’s a real-time node-based compositor built with JS and WebGL. It means you can:

  • blend and synchronize movies with dynamic data (other movies, pictures, 3D scenes…)
  • turn the compositing interactive, so driven by user behavior

In this example based on a green screen video footage, I’ve collected camera movements using Blender Motion Tracker Syntheyes, assigned them to a Three.js camera and synchronized the 3D camera with the initial footage. The compositing nodes created with Seriously.js are:

  1. Splitting the video source with crop effect (twice) to have the prepared color footage on one side and the greyscale outline on the other side.
  2. Making the source transparent using Luma matte with the Gradient Wipe effect (instead of chroma keying)
  3. Defining the Three.js canvas as source and put FXAA (Fast approximate anti-aliasing) on it.
  4. Blending the video with alpha channel and Three.js antialiased.
  5. Then blend the result with a solid colored background
  6. finally I’ve added few other effects, such as exposure and vignette.

Video source

Hollywood Camera Work is a nice place where find free green screen and tracking plates. The quality is pretty good and it’s perfect for tests.

Video preparation

Preparation process in AE

From the video source I’ve obviously removed the markers and quickly cleaned everything around the subject.
I’ve tried chroma keying with Seriously.js (i.e., rendered it with a green solid as background). It’s seemed logical, but the result is dirty. Click here to see my tests online: http://www.turbodrive.tv/arcade/webglmm_001/chroma.html. So instead I’ve prepared a greyscale version of the video used as luma matte. Both of the movies are in the same video file to prevent unsynchronized frames. As improvement, you can even smear the subject edges (on the top part of the source, not the part used as matte).

Video source – H264/WebM

Three.js animated camera

Using Syntheyes to extract camera movements, theses are tweaked in After Effects afterwards. Then camera data is exported and parsed in JS, and ultimately assigned to the 3D camera in Three.js

3D Scene – Three.js Canvas

Result: Composited video

Using Seriously.js, blending everything together and adding few adjustments.


Video and Three.js scene blended and rendered in Seriously.js canvas

Try it in a new window View the Javascript source

Compatibility and performances

Devices and browsers

It runs on desktop Safari, Chrome and Firefox Mac and Windows. I haven’t tried it on Opera and Seriously doesn’t target IE (actually the Three.js scene works on IE 11). Chrome v39 (and probably earlier versions) flips vertically the Three.js canvas. Read below to know more.

On tablets, it works on Safari 8 (iOS 8) but bunch of frames are dropped. Another thing is that you have to build your player in the way you play the video source by tapping the target canvas due to the video auto-play policy made by Apple. Take the standalone version as an example for this limitation: http://www.turbodrive.tv/arcade/webglmm_001/gradientWipe.html
I’ve quickly tried on Android tablet, it’s pretty similar to iOS (not smooth).

Performances improvements (Desktop)

Current video source framerate is 30fps. It’s a bit high and you can decrease the amount of calculations by using a video at 24 or 25fps. Currently both Chrome and Firefox reach the 30fps when playing the composited video. Safari 7 reaches 15fps on my computer when Safari 8 runs at 20fps only.

Canvas size is 640×360 and this is 640×720 for the video source including the luma matte. Performance mainly depends on the graphics card of the computer but it can be better by removing effects such as FXAA or simplifying the Three.js scene (removing lights and shadows). Another trick: for video such as my example with an isolated subject, the video source can be cropped and then re-placed with Seriously.js.
Anyway, it deserves more tests at higher resolutions and different framerates.

Chroma keying effect

Generally, chroma keying effect (example: here) solution needs less calculation as the framerate is higher (due to a smaller video source or the shader itself ?).
But in more than a dirty result, it works differently depending on the browser. For the same clipBlack value, the result is not the same between Chrome, Firefox and Safari. It works better on Chrome: with the same video source, it seems Safari and Firefox are replacing brighter (white) pixels by black pixels, no matter the property values you give to the chroma effect.

Pitfall and Code Tips

Three.js Focal Length

One of the difficulties has been converting After Effects camera focal length in the one used in Three.js. The key is basically that AE uses horizontal FOV when Three.js works with vertical FOV.
Here is the formula I’ve written to convert values for both 3D Cameras:

In the most common cases the film size doesn’t change, but take care of it. For example the camera solved with Syntheyes has a film width of 24.892mm.

Camera transformations parser

One way to get camera movements in Three.js is to export data from Syntheyes to AE and copy/paste the AE keyframes in a text file, then parse its content in JS. If you track the camera with Blender the process goes through AE too, since the Three.js exporter for Blender doesn’t export any data about the camera movements.

World axis are not the same so in Three.js rotationX, rotationY, y and z values are the opposite of the After Effects values.

Here is a quick parser I’ve made (I give it as is):

Instead of copy/paste camera data in .txt file you can also code a JSX script to export JSON file from After Effects. Follow the link to read some explanations about this kind of exporter in a previous post: http://www.turbodrive.tv/blog/after-effects-to-css-3d-workflow-part-2-transposition/#json-exporter

3D Camera and movie synchronisation

I’ve tried using playhead events of the video to update Three.js camera transformation but theses are not fired frequently enough. So it’s not accurate and totally unsynchronized. The solution is simply to base the updates on the Seriously.js render function.

Y-flipped Three.js canvas on Chrome 39

[Update for January 29th, 2015]

Since Chrome 40, there is no more y-flipped issue with Three.js used as canvas. The examples are updated but you can still continue to flip the source with the checkbox if needed.

Running on some version of Google Chrome (v39 currently), Three.js canvas is flipped vertically when defined as source in Seriously.js. This issue was reported here: https://code.google.com/p/chromium/issues/detail?id=420357
The bug does’nt exists on Chrome Canary (v41 at the moment) anymore and may disappear in a next version of Chrome Stable. Temporarily the script detects the browser and in the case of Chrome, Three.js canvas is vertically re-flipped with Seriously.js.

You can use a Transform node or just apply a flip transform when define the canvas as source:

It’s seems pretty simple but Seriously.js is not fully documented yet.

Tracking

The quality of the tracking is really important. It’s not only a question of video compression or resolution but it’s also related to the work of the camera operator. For a good camera match-moving you need more than a decent parallaxe on your footage or you’d spend hours tracking the camera movements.

Quick note about the software used for the camera tracking. I used to work with Syntheyes for few years. For this post I use Blender, which is free and provide tools for camera and objects tracking since the 2.62 version.
After all, I’ve never been really satisfied of the tracking made with Blender: to much errors when solving the camera, even with a lot of accurate trackers. I was probably doing something wrong?
So I went back track with Syntheyes. I was happy to see the quick results it gives with the auto-tracker. Much better than Blender. But I think the most interesting is the amount of tools when doing a supervised tracking and the available features for tweaking the solver and get more precise results.

Planar Tracking and Camera Tracking

The workaround here can be used with planar tracking instead of camera tracking. The planar tracking is pretty easy to do and works pretty well. Briefly, the workflow is:

Tracking in Mocha > Export to After Effects Corner pins > Exporting corner pins in JS > Parsing in JS and transforming image(s) in canvas.

In this case the keyed movie is not blend with 3D scene but with a projective texturing. Let me know if you’d like I publish a small example.

To conclude, I totally recommend Seriously.js. It’s an awesome lib. Even if it’s not fully documented yet it works pretty well, easy to use and very powerful. I expect a lot of crazy stuff made using it.

Thanks for having read this post. More experiments soon!