Overview

Despite recent impressive gains in realism, computer graphics is currently unable to effectively generate images of objects and environments that convey an accurate sense of distance and size. This problem is most apparent in the failure of computer graphics imagery to simulate the appearance of something that is really large.

While some of the difficulty lies in limitations of the various display technologies available to us, computer graphics imagery itself appears to lack critical visual cues to distance and scale. Although we understand that a problem exists, almost nothing is yet known about the nature of this missing information. Our research therefore focuses on both determining visual cues that are perceptually relevant for computer graphics and developing methods for rendering images that include these cues.

Accomplishing these goals requires new insights into the 3D information extractable from 2D images, modifications to graphics algorithms in order to better render salient information, and sophisticated perceptual experimentation to validate that people can actually perceive the intended 3D space. Our approach is intrinsically interdisciplinary, involving perceptual psychology, computational vision, and robotics, as well as computer science and computer graphics.

Accurate perception of size and distance from computer graphics is particularly important in immersive interfaces that aim to give a person the sensation of actually being in a virtual world that in fact only exists as a computer model. In such situations, veridical visual perception is required for tasks ranging from fine manipulation to locomotion through large outdoor spaces. This will require new computationally tractable ways to simulate illumination effects, material properties, and complex geometry in both man-made and natural environments.

Immersive interfaces provide the opportunity to explore the coupling between visual perception and action which is critical to human interaction with complex spaces. We believe that when a user is interacting with a computer simulated spatial environment, it is important that the user be able to walk through the environment and manipulate objects of interest in a manner that is as natural and realistic as possible.

Current Research and Open Questions

Lots of questions, few answers...

Visual Realism and Depth Estimation

Surprisingly (at least to us), it is still an open question whether or not visual realism aids in accurate distance judgments based on computer graphics imagery. A number of different research groups have now reported that even with visual immersion, appropriate linear perspective, and binocular stereo, distance judgments while viewing computer graphics are significantly underestimated compared to those made while viewing the "real world," at least for distances ranging out to 10m-20m. The effect is likely even more pronounced for longer distances, though we are unaware of formal studies confirming this.

  

Estimating distance in a virtual world

A computational analysis shows that there are actually only a few visual cues that can provide information about absolute depth in the 10m-20m range. We are exploring the hypothesis that one such cue -- familiar size -- depends on levels of visual realism that have not been used in prior experiments measuring depth judgments based on computer graphics.

Global Illumination as a Contact Cue

Renderings based on only local illumination often have a "cookie cutter" appearance, with objects tending to visually float in space rather than being seen as in contact with other objects and surfaces. It is well known that cast shadows are a visual cue to both the 3-D position of objects in a scene and to the fact that an object is in proximity to another object or an extended surface. The evidence is conflicting, however, on the accuracy with which shadows need to be rendered in computer graphics to achieve this effect. Understanding which aspects of shadow realism are perceptually salient is critical in order to balance effectiveness and efficiency in interactive applications.

  

Illumination cues for object contact

In computer graphics, interreflection is understood to be important for achieving subjective realism. Until recently, however, interreflection has not been investigated as a spatial cue. We have now shown that interreflection can function as a contact cue in much the same way as shadows. In at least some situations, barely perceptible interreflections can generate a visual sense of contact that is comparable in effectiveness to prominent shadows.

Interaction of Visual Perception and Locomotion

People naturally walk through the complex real world while maintaining spatial orientation in an effective manner. Traveling through virtual worlds using current computer interfaces is far less easy. We know that physical locomotion through an environment greatly assists in keeping track of where you are in the environment and where other locations are with respect to your own position. We are developing a locomotion interface with the goal of creating veridical visual and motor perception of distance, speed, and slope, and a realistic experience of turning and translating for accurate updating of locations in the environment. One key research question is whether we can build displays that synergistically combine visual information, generated by computer graphics, with biomechanical information on locomotion, such as generated by walking on a treadmill. In particular, can a user of such a system get a more accurate sense of a simulated space than would be possible using computer graphics alone?

Infrastructural Issues

In conducting experiments we have found ourselves making custom software to control the parameters of interest. For example, few if any packages allow one to create images with specific levels of direct lighting or interreflection turned on or off. We would like to develop a public-domain code infrastructure to make this process easier. Such an infrastructure should 1) avoid all discretization errors, and 2) have a clean and flexible code base that is easy for researchers to modify. We will discuss a plan to design such a framework, and seek guidance on what is needed in such a framework.