The goal of realistic image synthesis is to produce images that capture the appearance of real objects and scenes. Over the past 30 years, great progress has been made toward this goal with the development of physically-based rendering algorithms that accurately simulate the propagation of light in complex environments. However, even with these advances, generating visually realistic images is still more of an art than a science. One of the major problems is that while failures of realism are immediately evident, we don’t have good guidelines for achieving success. One of the tacit assumptions often made in graphics is that if physically accurate simulations can be generated then visual realism will follow. However physical accuracy is neither necessary nor sufficient to produce visually realistic images. The following paragraphs describe three standards of realism that that might help us define the criteria we need to meet to generate images that capture the appearance of the world around us.

The first standard is physical realism. Here the criterion for realism is that the image has to provide the same visual stimulation as the scene. If we neglect optical filtering and scattering in the eyeball, this means that the image has to be an accurate point-by-point representation of the spectral radiance values at a particular viewpoint in the scene. Although physically-based rendering algorithms can create accurate simulations of light transport, and these may be of great use in engineering and design applications, adopting physical realism as the criterion for generating visually realistic images has a number of important drawbacks. First, physically-based rendering algorithms are extremely computationally expensive, which limits their usefulness for many graphics applications. Second, even if the efficiency of the algorithms could be improved, it’s often impossible to accurately display their results, because the color gamuts and intensity ranges in the simulations frequently exceed the capabilities of existing display devices. Finally, in most cases, doing physically-accurate rendering is overkill if your goal is to make images for people to look at because of the limits of human visual processing.

These considerations lead to the second standard for realism and that’s photorealism. In computer graphics, photorealistic rendering typically refers to generating an image that’s indistinguishable from a photograph of a scene. This is a fine goal, but unfortunately it begs the question of realism because it doesn’t explain why a photograph is realistic. This is largely an unanswered question, but at least one approach is to restrict the scope of photorealism to photometric realism. Here the criterion for realism is that the image has to produce the same visual response as the scene even if the physical energy coming off the image is different from the scene. Adopting this criterion allows us to include the observer in the rendering process, and take advantage of the well known spatial, temporal, chromatic, and dynamic range limitations of vision to relax the precision we need to create visually realistic images. One example is standard color imaging which takes advantage of the trichromatic nature of color vision to create the appearance of a wide range of hues by combining three primaries. Another is the use of threshold visibility metrics to determine sampling parameters in perceptually-driven rendering algorithms. However, using photorealism as a standard for visual realism also has a number of drawbacks. First, although photorealistic rendering algorithms can be faster than physically realistic algorithms, they are still far too computationally expensive to be useful in many applications. Second, it’s not clear that photorealism is necessary or even desirable in a wide range of applications. Finally, adopting photorealism as the standard for visual realism in computer graphics categorizes most computer-generated images as failures, yet says nothing about their obvious utility.

These problems suggest a third standard for realism and that’s functional realism. Here the criterion for realism is that the image has to provide the same visual information as the scene. Information here means knowledge about the properties and relations of objects (e.g. shape, size, position, motion, material, etc.) that allows a user to make reliable visual judgments and perform useful visual tasks. Here realism is measured in terms of the fidelity of the information the image provides. If an image lets you do the task you need to do, and allows you to perform as well as you could in the real world, then for that task, the image is realistic. The beauty of a functional definition for realism is that it admits a wide range of rendering styles, from physically-accurate simulation, through photorealism, to more abstract approaches. One example of functionally-realistic images are the renderings used in flight simulators. These images aren’t physically accurate nor are they photorealistic, but they are functionally realistic because they provide the user with much of the same visual information they would get if they were flying a real plane. The proof of the realism of these images is that users can learn skills that transfer into the real world at high levels of performance.

An important direction for future research in realistic image synthesis will be to develop metrics that will allow us to integrate these different standards of realism into a coherent framework. One promising approach could be to conduct psychophysical experiments that explore the relationship between physical accuracy and visual fidelity in image synthesis. On the basis of these studies we should be able to develop efficient realistic rendering algorithms that adapt gracefully depending on the resources available and the demands of the application, balancing accuracy and efficiency, but always maintaining visual fidelity.