My current research interests lie in the development of inverse rendering algorithms for virtual and mixed-reality applications. In particular, I have recently been examining the problem of reconstructing geometry and illumination information from images of real-world environments. One particular application of these inverse illumination algorithms is in capturing illumination data for use in augmented reality applications, where synthetic objects need to be illuminated by the same light as the real objects in the scene.

Figure: Digital photograph of a real scene (left), and synthetic renderings using reconstructed illumination and material data (middle and right).

Accurately estimating illumination in general environments is a difficult problem. In many cases, such as outdoor or large indoor scenes, obtaining a complete geometric model is not possible. Additionally, capturing radiometric information (in the form of photographs) for each surface is a time-consuming task. The environment could also be illuminated by many different types of light sources, or a mixture of nat ural and artificial illumination. A flexible inverse illumination algorithm should work with partial geometric and radiometric models, and be able to handle arbitrary types and mixtures of illumination.

Because of the complexity of the task, any algorithm which will work robustly under these conditions must make simplifying assumptions about the way which illumination is captured and used. It is here that knowledge of the human visual system will be of great benefit, both in simplifying the process of capturing illumination data and accelerating the rendering of synthetic objects. If we require our augmented reality renderings to be visually accurate, rather than physically accurate, can we use this to ease the process of capturing illumination data? Interesting questions arise, such as how accurately must indirect illumination be represented? Do we need to know the exact position and size of each light-source in order to render accurate shadow boundaries, or will approximations suffice? Also, given the fact that we have limited resources available for rendering, which aspects of the illumination should take precedence when interactive frame-rates are required?

It is problems like these which I feel need to be addressed if visually realistic augmented reality is to be achieved. The issue of perceptually accurate illumination in mixed-reality applications has received little attention compared to other fundamental problems like tracking and image display, and I hope that discussions at this campfire will start to raise to some of these issues.