Warp mesh patch for Quartz Composer

The Quartz Composer patch discussed here warps an image (or movie) in an arbitrary way as specified by a user generated mesh stored in a simple text file. There are many potential applications for this but the motivation for this capability arose from a very precise requirement, namely the warping of images for projection into immersive spaces by using a projector and a spherical mirror.

This document assumes the reader is familiar with the topics listed above, namely the use of a spherical mirror as a low cost single projector solution for immersive displays. And that this technique requires image warping such that the result appears correct on the final surface. Additionally, the concept of navigable movies ... movies where each frame consists of more than a limited perspective projection and the viewer may change his/her view while the movie plays.

In all the above applications an image is warped (distorted) using a regular mesh consisting of vertex coordinates (x,y), texture coordinates (u,v) and an intensity mapping (i). The patch described here adds this warping support to Quartz Composer and is a direct analog to the existing stand alone application "warpplayer" (it uses the exact same warp mesh file format) except that it adds the enormous additional functionality of Quartz Composer to create more dynamic/interactive experiences. The rest of this document will concentrate on applications intended by the original designer of the patch, every attempt has been made to keep it general so that hopefully other uses will rise.

Description of inputs

Mesh file: file name of ASCII warp mesh, if the file name starts with "/" then it is an absolute file path, if it starts with a "~" then it is relative to the users home directory, otherwise relative to the current Quartz Composition directory. For portability reasons the last option is usually the preferred one. Texture control u, v: either clamp or wrap. X,Y,Z position: provided for generality, not used for the purposes discussed here. Width, Height: this controls the width and height of the panel the image/movie is displayed on. For the purposes discussed here one is always running in full screen mode, for the image/movie to fill the frame the width and height should be the inverse of the aspect ratio. So for 4:3 aspect the width and height should be 0.75, for 16:9 it should be 0.5625. Delta u,v: These modify the mesh texture coordinates of the warping mesh. Not all adjustments make sense, the most common for panoramic images/movies is to map the mouse movement to the delta u input to enable horizontal panning. Theta: This also modifies the texture coordinates of the warping mesh. For the applications discussed here it is mostly used for rotating a fisheye projection for planetarium content. Image: This is the input port for the image (or movie).

Examples

This is perhaps the most common type of application for planetariums using the spherical mirror projection technique. The input image is a fisheye and the only sensible navigation is rotation about the center of the fisheye. Note the use of the intensity mapping to compensate for the variable density of pixels, and hence light level on parts of the dome. Example: Moonlight courtesy Andrew Quinn Sample mesh file: fisheye.data

Spherical

This example also uses a spherical mirror and a single projector but it is for a dome orientated 90 degrees to a planetarium dome. The warp map below takes a full spherical projection (image or movie) and warps it for a 16:9 projector and spherical mirror. Varying "Delta u" allows one to pan left/right while the movie is playing ... hence a navigable movie. Example: Hawkesbury River courtesy Volker Kuchelmeister using the Ladybug camera Sample mesh file: spherical.data

Cylindrical

The following warps a cylindrical panoramic projection (image or movie) to a perspective projection. In the case of a movie made up of cylindrical panoramic frames this is another example of a navigable movie. Varying "Delta u" also achieves the panning left and right, adjusting "Delta v" is a vertical pan up or down. Example: Place-Hampi courtesy Sarah Kenderdine, Jeffrey Shaw. Sample mesh file: cylindrical.data

Planar

This last example is a large "standard" movie, a perspective projection. Here the mesh simply implements a 8 times zoom, varying "Delta u" and "Delta v" allows one to pan around within the larger movie. Example: "6dF: Beyond the Crux" by Paul Bourke Sample mesh file: planar.data

Download

Notes

• The patch has been tested so far with Leopard and QC3, as well as SnowLeopard and QC4. Noting however that for many movie intensive compositions QC3 is often much higher performing.

• It should be installed in "/Library/Graphics/Quartz Composer Patches" directory.

• The trick of course is how to generate the mesh file for a particular purpose. In some cases (eg: cylindrical to perspective) there is a clear mathematical relationship. In other cases, such as the spherical mirror based projections, one needs a calibration program where one aligns a known test pattern while adjusting the mesh to give the right transformation. This, meshmapper, is an example of such an alignment program.

• The mesh file format is quite straightforward. The first line of the file indicates the type of image format the mesh is designed for. For example: 1=planar, 2=fisheye, 3=cylindrical panorama, 4=spherical panorama. The second line indicates the dimensions of the mesh. The remaining lines describe each node of the mesh, they include the (x,y) position in normalised screen coordinates, (u,v) texture coordinates, and (i) a multiplicative intensity value.

• The image quality seems to vary significantly with codec. Photo-JPEG seems to work particularly well, Motion-JPEG seems to be particularly poor quality even though this isn't evident with the QTPlayer or warpplayer.

• There would also seem to be a bug in Apples QuickTime API for movies of width 4096 pixels, namely it mangles the 4096th column on nVidia cards but it works fine with ATI cards. In both cases the cards tested have a maximum texture size of 4096. Note that this happens in the QuickTime player from Apple, as well as warppatch.

Example: PufferSphere display balls

The Puffersphere displays employ a fisheye lens located at the radius of a semitransparent sphere. The exact geometry varies depending on the radius of the spheres and the projector being used. The challenge is to distort a spherical projection into the fisheye format such that the result on the sphere appears as a spherical projection should.

Sample input image from a video sequence of the LadyBug-3.

The resulting fisheye image, navigation is currently limited to rotations about the north pole (center of the image).

The QC composition. Note that this was additionally verification of pbmesh running with Snow Leopard and the associated new version of Quartz Composer. A simple example is given here: pufferfishpackage.zip. Note that this is designed for the 0.8m sphere, the field of view for the fisheye is different for the other radii versions.