Level of Detail Control for Real-time Computer Graphics and Virtual Reality: Applications of Eye-tracking.
In recent years there has been an increase in interest in the use of eye-tracking technology in real-time computer graphical applications such as Virtual Reality, games and simulations. Deering (1992) states that: "(eye-tracking) is a promising long term solution, since gaze direction can be exploited for other purposes such as identifying the region of screen space - corresponding to the foveal portion of the retina - that deserves to be rendered with high spatial detail."
When considering LOD control in real-time simulations, we are not only referring to the image which is rendered at each frame, but also to the simulation of the motions and interactions of the entities in the virtual environment. There are many opportunities to use information about the viewer's current region of interest to reduce the negative impact of reduced accuracy. For example, we can pre-process models of objects to ensure that perceptually important features are retained longer; When actually rendering scenes and objects, the viewer's current fixation point can be used to determine which parts of the scene to render at high detail; Similarly, the animation of objects and events which are currently being watched needs to be much more convincing than those happening peripherally.
In time-critical computer graphics applications, such as Virtual Reality (VR), three-dimensional objects are often represented as meshes of triangles. Real-time frame rates limit the number of triangles the graphics engine is able to display per frame. Therefore, meshes with a high number of triangles often have to be replaced by meshes with a lower number in order to achieve acceptable display rates; i.e. the level of detail (LOD) needs to be reduced.
Building the Mesh
A viewer's interest in a triangle can not be determined a priori. Hence the object in question is shown to a test viewer, while simultaneously their eye-movements are tracked. We associate a counter with each triangle to measure the frequency with which they fixate it. The number of fixations is then interpreted as a measure for the test viewer's interest in a triangle, and the order with which regions are coarsened is updated accordingly. In other words, the lower LOD meshes get moulded interactively by the test viewer's interest. Meshes created in this way showed significant improvements over meshes created by more traditional methods, even when the LOD was significantly reduced. Striking results could be achieved within seconds, depending on the number of triangles and the topology of the object.
Figure 2: A Terrain Mesh.
View-dependent Continuous Level of Detail
When rendering large meshes on the computer, both storage and rendering time can be problematic. Even if the memory and power were available, the time spent rendering triangles that don't make a perceivable difference would be better spent improving the image quality of triangles that do. When an eye-tracker is used, detail can be added wherever the user is looking. This will give the impression that the mesh is much more detailed than it actually is. In the specific problem of rendering detailed terrain, there are several LOD techniques available (see Figure 2). An existing method called ROAM, presented by Duchaineau et al. 1997, has been adapted to incorporate foveation. The focus of the viewer's attention is assumed to be the intersection between the line from the viewer's position in the direction of gaze, and the mesh. The distance of this intersection point from the viewer is used to determine how big a section of the terrain needs to be drawn in higher detail. Similar approaches may be used in any application which renders objects modelled by meshes.
Simulation Levels of Detail
In order to create believable animation of objects and characters, the physics of the real world need to be taken into account. In particular, when objects come into contact with each other, a suitable response must occur. However, often it is not possible to perform the computations which will enable an entity to react in a realistic way to un-predictable events. Firstly, simply determining whether they have collided or not is expensive. For a highly accurate response to be performed, the contacts between the objects need to be modelled in detail, and finally the physical response must be determined. Again, an eye-tracker can indicate which collisions are the most important and this information is then used to schedule the processing of such events. A hierarchy of spheres is used to approximate the topology of the object, as in Hubbard 1996 (See Figure 3). When the time available for collision processing has expired, response occurs at the resulting level of detail.