Panel: Evaluation and Experimental Design

Kirsten Cater, Computer Science, Bristol University

Jim Ferwerda, Program of Computer Graphics, Cornell University

Carol O’Sullivan, Computer Science, Trinity College Dublin

Holly Rushmeier, IBM T.J. Watson Research Center.

Ben Watson, Computer Science, Northwestern University

• Why evaluate?
  • Validation: prove we’ve “done good” (passive)
  • Improvement: the evaluation process is designed to elicit new knowledge. (active)
• Where to start?
  • Apply existing principles (e.g. models), vs.
  • DIY user studies
• In the past, when people applied perceptual principles to evaluate their results, observations were often either:
  • Too superficial (folklore… just an aura)
  • Sound principles, applied inappropriately
  • Sound principles, applied appropriately

If we apply existing models or use data from experiments already carried out, the following issues may arise:

• Do we really need to mimic the brain?
• Stimuli too simple (dimensions too low)
• Tasks too specific
• Can’t always combine results for more complex scenarios
• Results often contradictory
• Sometimes too few subjects
• Results often
  • Not robust (i.e. only work under highly specified conditions)
  • Too expensive to compute “on the fly” for real-time applications.
• Big learning curve needed for non-psych researchers

If we accept that our model does not have to mimic the brain completely accurately, it is often preferable to derive a empirical model based on DIY user studies.

• Pro: You can tailor your experiment to the specific task you're interested in
• Cons: This process will be much slower than simply re-using existing models, and there is also a lack of standard requirements for carrying out perceptual experiments for use in the field of Computer Graphics

The following issues can arise when designing psychological experiments in the field of Computer Graphics:

• Avoiding “dimension explosion”
• “Bad” participants: Discard or Include?
• How to devise a natural task
• Should one examine factors in isolation, e.g. incrementally or in a top-down fashion, requiring participants to provide a subjective opinion: e.g. I like it/not.

Further issues that were discussed included the use of eye movements as an evaluation tool, and the advantages of real vs. simulated experiments. Such experiments in dynamic situations was recognised as being particularly complex, but worth pursuing.

Contact