Fundamental to visualization is the idea of recruiting the human visual system and its mechanisms, and so it is unsurprising that there this a body of work in this area. However, much of the work to date has focused on comparatively low-level issues, such as the use of colour, the role of lighting, and the effects of transparency. The role of higher levels of cognitive information processing are still largely unexplored in this domain. So while, for example, Hibbard et al. [HLH+95] acknowledge the importance of feedback between "bottom-up" and "top-down" processing in perception, most of their concerns, such as the need for responsiveness during interaction, are actually generic and say little about how the information in the display is being extracted, used or understood. Similarly, Rheingans and Landreth [HL95] concern themselves with issues that apply mostly at the pre-attentive or early phases of visual information processing.

In visualization, as in other areas of interaction between humans and computers, research has concentrated largely on the development of practical tools and techniques, leaving a comparative dearth of theory about how particular visual representations operate to mediate understanding and exploration. Although analyses can be carried out on specific systems, we are a long way from having a detailed theory that would, for example, guide the development of techniques for visualising novel domains or addressing trade-offs in methods for working with large-scale data in abstract spaces (for example, graph visualization [HMM00]).

An important step in describing visualization, and how visualizations work, is a conceptual separation of the underlying data domain, the structure of the representations used to visualise that data, and the operations that can be performed on those representations. Mention should be made here of the work of Casner [C91], Zhang [Z97], and Chi and Riedl [CH98]. However, although research such as Casner's acknowledges the role of cognitive processing in using visualization systems, it essentially treats the cognitive system as a "black box". Fundamental constraints and capabilities of the human to process and respond to information presented by the system are left implicit, or at best treated as basic assumptions.

A key part of the problem is that unpacking the cognitive box, and relating its contents to the concerns of visualization, is a task that requires multidisciplinary research. And the disciplines involved, computing on the one hand, and cognitive psychology on the other, have traditionally operated with quite different models and theories. Within HCI, a number of "bridging representations" have been developed to try and integrate the products of multi-disciplinary analyses [BD97], but at the cost of separating results from modelling theory.

Over the last six years, the author of this position paper has been working with three other researchers -- David Duce from Oxford Brookes University, whose background is in computer graphics, distributed systems, and formal techniques, and two psychologists, Jon May from the University of Sheffield, and Phil Barnard from the Medical Research Council's Cognition and Brain Sciences Unit in Cambridge, on an approach to building theories of interaction between computating devices and an explicit account of human cognitive processing. This work, which is called "syndetic modelling", represents one possible bridge between the understanding developed within computer graphics of techniques for building interfaces, and the body of knowledge within psychology about human cognitive resources and processing constraints.

Our approach has two main elements. First, as our cognitive model we utilise ICS (Interacting Cognitive Subsystems), which provides a distributed architecture for human information processing and embodies constraints about the flow and nature of processing that can be carried out [BM95,M00]. In contrast to small-scale "local" theories, for example of visual adaptation or colour perception, ICS deals with larger-scale patterns of activity, while providing place-holders in which specific local theories can be situated. Figure 1 gives a flavour of the model in the context of interacting with a system via gestural input recognised by data glove. In ICS, cognitive processing is distributed over nine distinct subsystems, each specialised to deal with a particular level of representation. In the figure, top down goals directing gesture formation are originating at a "propositional" level (1) and are then mapped through a level of spatial orientation (2) to coordinate limb control (3) and hand position (4). In parallel, a bottom up stream derived from visual input (5) is being used to support this activity by feeding back a propositional awareness of system state and gesture recognition (6). A third stream of information carries proprioceptive feedback (7) from the hand back to coordinate limb control (8).

Figure 1. The ICS architecture (from [D95])

The second element of our work is the observation that to apply results from cognitive modelling to the design of computing systems, we need a common framework in which we can study how information is generated, exchanged and processed by both sides of the interaction. Figure 2 shows how the cognitive processes described above are situated in a larger system of processes including those responsible for gesture recognition and rendering. Rather than attempting to simulate this exchange, we have used mathematical modelling to specify the behaviour of both cognitive and computing systems, and then explore the possible behaviours allowed by the conjoint system.

Figure 2. Information flow between computing and cognitive resources (from [D95])

Applications of this technique to date have been in the area of multimodal interfaces [DBDM99] and novel interaction techniques [D95]. However, we believe that the same approach could provide a useful tool for understanding visualization, and in particular the design of representations for abstract information. For example, Duce and Duke [DD95] argues that this kind of framework has the potential to support arguments about whether the reasoning afforded by particular visual representations is "truthful" with respect to the underlying domain.

This line of work presents a number of challenges. For example, how can the information carried by visual representation be characterised in a way that can be related to the cognitive processes that operate on these representations? Building on Casner’s work [C91] linking "logical operations" on abstract data with "perceptual operations" on presented data, [DH95] developed an approach of using mathematical structures to characterise the presentations. There are relationships between these structures and the representation of information within ICS, however much remains to be done to develop the theory to a point that it can address interesting questions.

An example of the kind of design question that is of interest is represented by work on tree and graph visualization. Figure 3 shows a tree structure visualised using a tool called "Latour", developed by Ivan Herman and colleagues at CWI Amsterdam. One issue that is being explored within the visualization community is how to render such structures once the scale of the data set increases significantly. In the case of tree-like structures, one solution explored in [HMM+99] and shown in Figure 4 is to develop metrics for identifying the "importance" of sub-trees, and providing a simple means of systematically eliding sub-trees by introducing a "skeletal" representation of the elided detail. This approach though raises the question of how effectively users will be able to navigate and understand such overviews, and indeed how such overviews might be modified or tailored to support specific tasks in the application domain. More fundamentally still, can such techniques be generalised beyond trees to more general graph-like structures, and what are the cognitive implications of doing so?

Figure 3. Viewing a tree in Latour

Syndetic modelling is still at an early stage of development, and much remains to be done, both in terms of the capturing more of the underlying cognitive theory, and in understanding how to represent interactions such as those between a visualization system and its user. It may yet be that such a theory does not provide the right insights to understand many of the questions about the design of visual representations and visualization interfaces. But the questions raised by visualization are interdisciplinary, and its only by building bridges between the disciplines that deeper, long term results will emerge.

Figure 4. Skeletal view of a tree

Acknowledgement

Figures 1 and 2 are reproduced from reference [D95] by kind permission of the European Association for Computer Graphics.

References: