How do we evaluate perceptually-based methods in computer graphics? A difficulty in making progress in applying perceptual science to graphics is that the assessment of the value of a method is different from what graphics researchers are used to. We can't prove that we reduce the computational complexity of an algorithm, our goal is not to produce dramatically different visual effects, and we can't use a physical instrument to measure the fidelity of our simulations with respect to physical reality.

There is a history in computer graphics of applying perceptual principles ranging from general observations to the use of very specific data. In some cases the observations border on folklore, and other cases they are very sound. In some cases the specific data are applied appropriately, in other cases they are used to give the aura of scientific results to a questionable application. Simply stating that perceptual principles are used doesn't establish the utility of a method. We need user studies, but what are the requirements for these studies? Anyone who has tried to publish these results know the criticisms that are returned -- there should be more subjects, the subjects should be a cross-section of society (not just computer scientists), more variations should have been studied, and what about this, that and the other thing. On the other hand, there have also been publications where very poor studies are published, just because reviewers were so glad to see any effort at validation. Since the field of graphics has no standards or examples for what is adequate, publishing results is a hit or miss proposition.

In graphics we tend to want to solve everything now. There is a desire for a comprehensive method that can resolve substantial issues such as how should an object be represented or what is important in an image. In general, progress is much slower, and involves many experiments examining many subtle issues. How great a contribution is enough to warrant attention?

Clearly we need to look to psychology for guidance on conducting studies. That needs to be combined with our own history of what constitutes a useful contribution to computer graphics.

In the past there was a problem in publishing systems papers in computer graphics, because systems inherently do not use new algorithms, but creatively combine existing techniques. The community has worked on establishing what the criteria are for a systems paper, e.g. documentation of the decisions that were made in the design, report of experience in using the system. Similarly we need to establish some guidelines on how do we as a community evaluate new methods that apply perceptual principles to computer graphics.