Manual and Cognitive Benefits of Two-Handed Input:
An Experimental Study
Andrea Leganchuk, Shumin Zhai,and William Buxton
AbstractOne of the recent trends in computer input is to utilize users’ natural bimanual motor skills. This paper further explores the potential benefits of such two-handed input. We have observed that bimanual manipulation may bring two types of advantages to human-computer interaction: manual and cognitive. Manual benefits come from increased time-motion efficiency, due to the twice as many degrees of freedom simultaneously available to the user. Cognitive benefits arise as a result of reducing the load of mentally composing and visualizing the task at an unnaturally low level imposed by traditional unimanual techniques.
Area sweeping was selected as our experimental task. It is representative of what one encounters, for example, when sweeping out the bounding box surrounding a set of objects in a graphics program. Such tasks can not be modelled by Fitts' Law alone (Fitts, 1954) and have not been previously studied in the literature.
In our experiments, two bimanual techniques were compared with the conventional one-handed GUI approach. Both bimanual techniques employed the two-handed "stretchy" technique first demonstrated by Krueger (1983). We also incorporated the "Toolglass" technique introduced by Bier, Stone, Pier, Buxton and DeRose (1993). Overall, the bimanual techniques resulted in significantly faster performance than the status quo one-handed technique, and these benefits increased with the difficulty of mentally visualizing the task, supporting our bimanual cognitive advantage hypothesis. There was no significant difference between the two bimanual techniques.
This study makes two types of contributions to the literature. First, practically we studied yet another class of transaction where significant benefits can be realized by applying bimanual techniques. Furthermore, we have done so using easily available commercial hardware in the context of a task common to graphical interaction. Second, theoretically this study contributes to our understanding of when and how to use bimanual interaction techniques in future systems and why bimanual interaction techniques they have an advantage over unimanual techniques. A literature review on two handed computer input and some of the most relevant bimanual human motor control studies is also included.
Categories and Subject Descriptors: H.1.2 [Models and Principles]: User/Machine Systems - human factors; H.5.2 [Information Interface and Presentation]: User Interfaces-input devices and strategies, interaction styles; I.3.6 [Computer Graphics]: Methodology and Techniques-interaction techniques;
General Terms: Human Factors, Experimentation, Design, Measurement.
Additional Key Words and Phrases: Input devices, Two-handed input, Bimanual input.
Introduction
Contemporary GUIs have come to be known as Direct Manipulation systems (Shneiderman, 1983). Correspondingly, the research described in this paper is based on the premise that one way to improve such systems is by extending the richness of what can be manipulated and the directness with which it can be done.
One approach to doing so is to develop interaction techniques that take advantage of the user’s potential to continuously coordinate both hands in the performance of a broad range of tasks. A number of systems have demonstrated how this potential can be employed in human-computer interaction (Buxton and Myers, 1986; Sachs, Roberts and Stoops, 1991; Bier et al., 1993; Hinckley, Pausch, Goble and Kassell, 1994; Kabbash, Buxton and Sellen, 1994). While these examples have demonstrated the effectiveness of such techniques, they have not yet penetrated into everyday applications. One reason for this may be the overall cost in equipping systems with input devices which afforded this class of interaction. However products such as, the multi-device sensing capability of the digitizing tablets from Wacom (Wacom, 1994), for example, address this — at least for graphical applications.
The other issue has to do with developing a better understanding of where, when and how best to use two-handed techniques, and why they work. It is therefore important to review, augment and consolidate the relevant literature. The remainder of this Introduction reviews some of the representative bimanual prototypes, and then discusses some pertinent experimental and theoretical studies in the literature. TWe then present the central hypothesis of the current work is then presented — the motor and cognitive benefits of two-handed input.
The representative prototypes of two-handed input
Two-handed input, per se, is not a new topic in user interface technology. For example, typing on a QWERTY keyboard uses two hands, but does not fall into the class of interaction studied in this paper.
Typing can be characterized as a coordinated bimanual activity where each hand is performing a series of discrete tasks. While one of the most effective ways of entering text into a computer, this does not reflect the more continuous nature of most everyday bimanual actions.
Sketchpad (Sutherland, 1963), is an early example of a system that not only used both hands, but did so in a way that went beyond the discrete types of actions seen in typing. Users drew directly on the computer screen with a light pen using one hand, and could modify the pen’s action with the press of a button using the other hand. The buttons were used in conjunction with the pen in this manner to create, move, copy or erase straight line or circular shapes. This is a very early example of a two-handed input interface where the left and right hands performed highly integrated actions.
Another early use of this type of two-handed input was in 1982 when the Xerox Star computer system was introduced (Smith, Irby, Kimball, Verplank and Harslem, 1982). In addition to the unique new graphical interface, the designers used a set of buttons on the keyboard to add functionality. These early ‘function-keys’ were positioned on the left side of the computer’s keyboard for easy access by the left hand, while the mouse was held and manipulated with the right hand.. The user could select an object with the mouse and then use one of the function keys to specify the action to be performed. For example, to move a file one would select the file’s icon, depress the "Move" button and then select its new location.
Video games are by far the most common example of two-handed input. A typical game controller has a set of buttons on one side of the pad and a joystick (or a rolling button or set of four cursor keys) on the other side. Typically the buttons launch bombs or cause the game character to jump while the joystick controls the character’s directional movement. The game character’s defense often consists of having to keep moving to avoid getting killed while at the same time the launching offensive attacks like dropping bombs. This requires skill and good coordination of the two hands.
The last three examples all have the similar property that one hand performs a discrete task (pushes buttons) while the other hand performs a continuous task (moves a pointing device). A significant difference of the interfaces is that some require the two hands to operate sequentially (Star) while others require coordination between the two (video games).
Krueger (1983) contended that users should be able to manipulate graphical objects like objects in the physical world. He demonstrated that this was possible for a range of tasks, using both hands to stretch, position and rotate boxes and other graphical objects in an intuitive and natural manner. This was perhaps the earliest example of coordinated and integrated bimanual interaction, where both hands were performing continuous, rather than discrete, tasks.
Sachs, Roberts and Stoops (1991) developed a two-handed system, 3-Draw, for 3D CAD applications. With their system, a user held a pair of six-degree-of-freedom input devices, mounted on a light hand-held tablet, or palette, and a stylus respectively. The palette corresponded to the base (construction plane) of a 3D object being created and the stylus was used as a drawing and editing tool. The designer manipulated the palette in the real world to rotate and translate the object in the virtual world. The stylus was held in a position relative to the palette and used to sketch the object’s curves.
Hinckley et al. (1994) designed a 3D two-handed input system using passive props. In order for neurosurgeons to specify a slice of the brain data to view, the user manipulated two props, a head prop and a cutting plane prop. The position and orientation of the props could be sensed by the computer. By placing the plastic sheet on the doll’s head, the surgeon could control what "slice" of the brain was displayed.
In 1993, Bier et al. developed a new bimanual interaction technique called Toolglass. This employed an interface widget that behaved as though it were a transparent sheet of glass upon which resides tools which can be used within an application The Toolglass sheet’s position was continuously controlled by the non-dominant hand, while the dominant hand controlled the system cursor. The Toolglass sheet lay between the cursor and the graphical objects on the "desktop." To use a tool on the Toolglass sheet, the cursor was positioned over the object to be affected, and the Toolglass sheet positioned such that the desired tool lay between the cursor and that object. One could then ‘click through’ the tool to cause it to affect the object below. Since the Toolglass is transparent, both the tools and the application behind can be seen at the same time.
Hinckley et al. (1994) designed a 3D two-handed input system using passive props. In order for neurosurgeons to specify a slice of the brain data to view, the user manipulated two props, a head prop and a cutting plane prop. The position and orientation of the props could be sensed by the computer. By placing the plastic sheet on the doll’s head, the surgeon could control what "slice" of the brain was displayed.
For the rest of this study, we shall be concerned with the general class of bimanual interaction introduced by Krueger and further developed by Bier et al. Sachs et al., and Hinckley et al., that is, systems where both hands work in a continuous, rather than discrete, manner, and where they are coordinated and integrated in the performance of a particular "direct manipulation" task.
Empirical studies and theories of two-handed input
While the list of bimanual prototypes is fairly long, the converse is true for the list of relevant empirical studies.
Dillon, Edey and Tombaugh (1990) conducted an experiment which examined performance in another compound task. They compared five interaction techniques for a task where the user would select from a menu and then draw a line elsewhere on the screen. Of the tested techniques, one was a standard GUI one-handed method where one cursor would be switching between menu selection and drawing in order to complete the task. They also included two variations of a "two cursor" method which employed the use of both hands, one controlling a cursor over a menu for selection and the other for drawing (the menu size varied for the two conditions). In this experiment a small but insignificant advantage of two-handed input was found over one-handed input.
In 1986 Buxton and Myers performed experiments which evaluated the benefits of two-handed input for compound tasks. They performed two experiments, one on a positioning-scaling task, the other on a navigation-selection task. Both experiments had the property that both hands were performing continuous tasks. In addition, the two hands could move either in parallel or sequentially while performing either experimental task.
The positioning-scaling experiment was designed such that the left hand scaled an object while the right hand positioned it. The results showed, without encouragement, all but one of the participantsubjects performed some of the task in parallel and the level of parallelism correlated to performance improvement. Because participantsubjects spontaneously and almost immediately exhibited the ability to perform both tasks in parallel, the experimenters concluded that no significant additional cognitive load was imposed by the utilization of two-handed input.
In the second experiment conducted, the left hand navigated through a document (controlled the scrolling) while the right hand selected designated pieces of text. This experiment was performed by both novice and expert users using both unimanual and bimanual techniques. Although little parallelism was exhibited, the two-handed technique was still significantly faster than the traditional one-handed technique. This was due to the fact that no switching was needed between scrolling and selecting in the two-handed technique — each hand being effectively in "home position" for its own particular task.
The two experiments by Buxton and Myers established the first empirical evidence that the use of two hands in human-computer interaction resulted in higher motor manipulation efficiency without, as explained above, imposing significant additional cognitive load.
Dillon, Edey and Tombaugh (1990) conducted an experiment which examined performance in another compound task. They compared five interaction techniques for a task where the user would select from a menu and then draw a line elsewhere on the screen. Of the tested techniques, one was a standard GUI one-handed method where one cursor would be switching between menu selection and drawing in order to complete the task. They also included two variations of a "two cursor" method which employed the use of both hands, one controlling a cursor over a menu for selection and the other for drawing (the menu size varied for the two conditions). In this experiment a small but insignificant advantage of two-handed input was found over one-handed input.
In contrasting the results of Buxton and Myers with those of Dillon, Edey and Tombaugh, we must conclude that the benefits of two-handed input are task and design dependent. Clearly a deeper level of understanding of human bimanual function is needed for the successful design of two-handed computer input.
Various human motor control theories on the relative performance of two hands have been proposed and tested (e.g. Flowers, 1975; Todor and Doane, 1978; Schmidt, Zelaznik, Hawkins, Frank and Quinn, 1979). Many of these studies on bimanual action have demonstrated the co-operative nature of the two. It is a common observation that it is difficult to rub ones stomach while tapping ones head. The difficulty lies in the independence of the two tasks. A common experimental paradigm used in two-handed bimanual control is production of polyrhythm: two conflicting but isochronous sequences. Most people have great difficulty in coordinating the two hands in such tasks in which the two rhythms are not integer multiples of each other. Only skilled musicians following extensive training are able to accurately perform complex polyrhythms with ratio of 5:3. Studies showed this was possible only when participants adopted an integrated motor organization in which movements of the slow hand were subordinate to movements of the fast hand (Summers and Kennedy, 1992). Furthermore, simultaneous independent timing for the two hands could not be achieved even when parallel control was encouraged by training the participants at producing the required tapping frequency with each hand independently first. The interdependence of bimanual movements has also been found in many other studies. For example, Peters (1985) showed that even for skilled participants, there were severe resource limitations in terms of providing two independent streams of timed signals for the initiation and termination of movement in the two hands.
However, tThe one that is most helpful bimanual action research in understanding the conflicting results seen in the two studies discussed above that can be directly applied to two handed computer input is Guiard’s kinematic chain (KC) theory (1987). According to this analogical model, the two hands function as serially assembled links, with the left (or non-dominant) hand as the base link and the right (or dominant) hand as the terminal link. Fundamental to this theory is the cooperative and asymmetric nature of the two hands. Four basic characteristics of the KC model are (Guiard, 1987; Guiard and Ferrand, 1996in press):
Guiard’s KC theory of bimanual function is also reflected in the design of the Toolglass technique (Bier et al., 1993), discussed earlier. Thus, we should expect it also to result in significant improvements in performance.
The technique was empirically evaluated in a study conducted by Kabbash, Buxton and Sellen (1994). Their experiment compared four different interaction techniques for the task of drawing colored line segments between dots. This task required a menu selection of the correct color and then a drag to create the line from one dot to the next. The techniques included:
Finally, it is worth notingworthwhile to be reminded that the results also show that two hands are not always better than one. As in the Dillon, Edey and Tombaugh study, the two-cursor technique performed poorly — in this case (insignificantly) worse than the one-handed technique. So, while there are benefits to be gained, it is important that we undertake the research that will help us better understand how to obtain them.
Although most relevant to the current study, Guiard (1987) is but one approach to the study of human bimanual motor performance in the basic human motor control literature. Other studies on bimanual action tend to focus on elemental task experimentation. The co-operative nature of the two hands, however, was also be demonstrated at such a level. It is a common observation that it is difficult to rub ones stomach while tapping ones head. The difficulty lies in the independence of the two tasks. A common experimental paradigm used in two-handed bimanual control is production of polyrhythm: two conflicting but isochronous sequences. Most people have great difficulty in coordinating the two hands in such tasks in which the two rhythms are not integer multiples of each other. Only skilled musicians following extensive training are able to accurately perform complex polyrhythms with ratio of 5:3. Studies showed this was possible only when participantsubjects adopted an integrated motor organization in which movements of the slow hand were subordinate to movements of the fast hand (Summers and Kennedy, 1992). Furthermore, simultaneous independent timing for the two hands could not be achieved even when parallel control was encouraged by training the participantsubjects at producing the required tapping frequency with each hand independently first. The interdependence of bimanual movements has also been found in many other studies. For example, Peters (1985) showed that even for skilled participantsubjects, there were severe resource limitations in terms of providing two independent streams of timed signals for the initiation and termination of movement in the two hands.
Having reviewed the literature on two-handed computer input and some the most relevant bimanual human motor control studies, we now tend to the thesis of the current study, which offers a complimentary perspective to the existing views of bimanual computer input. While Guiard’s KC theory of bimanual action offers a helpful framework for designing two-handed input techniques, particularly on the asymmetrical division of labor of the two hands, it does not explain all the benefits of two handed input.We found that performance improvement with two-handed input often could not be explained at the motor-level alone. Part of it had to be attributed to cognitive factors.
Motor and cognitive advantage of two-handed input
Studies such as Buxton & Myers and Kabbash, Buxton & Sellen show that improved time-motion efficiency can be achieved using appropriate bimanual techniques. However, can the benefits be attributed simply to improved motor-sensory efficiency, or are there cognitive reasons as well?
One of the basic tenets of the acquisition of cognitive skill is the notion of chunking (Simon, 1974). Anderson (1980; 1982), for example, describes the acquisition of cognitive skills as based upon the compilation and proceduralization of knowledge into chunks formed around underlying sub-tasks. Likewise, Card, Moran & Newell (1980) describe users’ methods in the Keystroke-Level Model as being
Applying the notion of chunking and phrasing to (graphical) manipulation at the computer interface, Buxton argued that the physical affordances at the motor level of interaction could be designed to be compatible with and reinforce the desired chunking of sub-tasks at the cognitive level. He suggested that there were two types of "glue" that could be used as the catalyst to this aggregation: muscular tension and continuity of motion ("kinesthetic continuity"). Simply stated, he argued task elements that should be conceptually chunked should be physically chunked into a single gesture, or "phrase" (where, like in music, tension and movement are the key elements of establishing structure).
A common example of chunking is seen in making a selection using a pop-up menu in a GUI. Another example is illustrated in the common proof-reader’s symbol for ‘move’. In both cases, a number of sub-tasks are "chunked" into a single aggregate, or "phrase." (And, as in music, the phrase holds the elements together into a connected flow which is not generally interrupted.) In the case of the pop-up menu, muscular tension of holding down the mouse button is the glue that binds the elements of the phrase, whereas in the proof-reader’s move symbol, the bond is the kinesthetic continuity associated with articulating the mark of the symbol. While syntactically rich, due to the phrasing, each of the example transactions is perceived as a single gestalt.
From this perspective, we can think of the acquisition of cognitive skill as relating to the compatibility between motor action and aggregation of cognitive unit tasks. Conversely, one could argue that some problems in skill acquisition are a consequence of a lack of compatibility with the pragmatics of how actions are articulated. This was the perspective from which we investigated the potential cognitive benefits of bimanual input. Our basic hypothesis was that appropriately designed bimanual techniques should chunk better at the motor-level, and therefore be more compatible with the mental model of the user, as acquired from a lifetime of living in the everyday world.
Figure 1: Using traditional unimanual techniques, moving Rectangle A to Rectangle B requires (multiple) iterations of translation, rotation, and scaling. In contrast, using a bimanual technique that assigns two hands to controlling two opposing vertices, all of the three aspects can be "chunked" into one integrated process, which is closer to how one naturally views such a task.More specifically, using traditional unimanual input techniques, many graphical manipulation tasks, such as to move, scale and rotate an object (e.g. a rectangle), are forced to be executed on an elemental level (one aspect at a time with iterations), which impose unnecessary cognitive burden on the user (See Figure 1). Conversely, two-handed input may allow the user to perform at a natural level of chunking that corresponds to daily activities, possibly resulting in the following interrelated cognitive advantages:
Experiment 1
Task Selection
We had three criteria in mind when we chose the task for our experimental study. First, we wanted to select tasks that are practical in real computer applications so that the results of our experimental study can be applied to designing practical interaction techniques. Second, we wanted to choose a task that has not been extensively studied in the HCI literature so that the empirical findings in regards to two-handed input can be established on a broader base. Third, we intended to select a task that has the testing power to accept or reject our hypotheses (discussed in the section, The Hypotheses).
Studies in computer input have concentrated on "point-click" type selection tasks. This type of selection can generally be modelled by Fitts' Law (Fitts, 1954), and is fairly well understood. A form of graphical selection which has seen widespread use, yet is far less studied, is the sweeping of a bounding box around an area or group of objects. Figure 2 shows an example where the user is attempting to place a minimum bounding box around just the woman in the picture. This issue arises in both computer drawing packages as well as painting programs. Users often must select accurately a specific area of their drawing so as to include some parts in their selection while excluding other parts. This proves to be a difficult task when these parts lie close to one another but distant from the location of the beginning of the drag.. This class of transaction is very common and is representative of a larger class of graphical interactions. It has not been previously studied yet has been observed to be highly prone to user error.
Figure 2:Generalized example of the bounding box issue. The user is currently dragging out a selection box. The initial mouse press at the top left was incorrect (they are including part of the man’s arm unintentionally).The experimental task used in our study is an abstraction of the above area sweeping task. The participantsubjects were asked to minimally enclose the object presented with another figure. Six different basic geometric forms were chosen for presentation: squares, rectangles, circles, ovals, equilateral triangles and isosceles triangles. Two shapes were used as the encompassing figure, a rectangle and an oval.
ParticipantSubjects were directed to use an oval to enclose triangles, squares and rectangles. Similarly they were directed to use a rectangle to enclose circles and ovals. Figure 3 illustrates some examples of correct stimulus-response (i.e. target figure-encompassing figure) pairs.
Each encompassing figure has four control points, represented by the black square handles, which appear when the input device is released. The control points are points which by moving them, one can change the shape of the resulting figure.
Once the encompassing figure was drawn, if it did not meet criteria, a one screen pixel maximum distance between stimulus and response edges, then the user would have the opportunity to readjust the figure until it matched by reacquiring any of the four control points.
If the user missed the handle, the handles disappeared, the figure became ‘de-selected.’ The user re-selected the encompassing figure by clicking anywhere within the figure.
If the user chose the wrong encompassing figure tool (such as the rectangle tool when presented with a square) the system beeped and would not activate the tool. The user would then attempt again to choose the correct tool.
The user performed the task with a Wacom Tablet holding a stylus in their right hand and a 4-button puck in their left. This apparatus is described in detail in the section to follow entitled The Apparatus.
The Techniques
In this experiment we study three techniques using the same hardware framework. The only differences presented to the user will be in the form of the technique’s specific properties as outlined below.
Conventional One-handed Technique (C1)
This unimanual technique was modelled on the techniques and methods currently employed in traditional graphical user interfaces (GUIs).
The tools (encompassing figures) are available on a tearoff palette which may be moved by dragging its header frame to a new location. A tool is chosen by clicking on the palette on the desired tool. The tool will highlight to show it has been selected and the cursor, when over the drawing surface, will change from the default arrow to a drawing cursor. The user may then drag on the drawing surface to use their chosen tool to create the encompassing figure.
Upon release, the standard handles found in most drawing applications appear at the four control points of the encompassing figure. Adjustments may then be made to the figure by dragging these handles to new locations. See Figure 4 for an example of an encompassing figure with handles.
This technique takes advantage of using both hands to manipulate the encompassing figure based on a technique initially introduced by Krueger (1983). The tool is selected from a standard GUI tearoff menu as described above for the unimanual technique. The tool will highlight to show it has been selected and the cursor, when over the drawing surface, will change from the default arrow to a drawing cursor.
When the user presses down on the drawing surface the location and size of the encompassing figure is controlled by both hands. A cursor appears at opposing corners of the figure, indicating the points from which the left and right hands can "stretch" it. The left and right hand cursors are both stylized arrows but are oriented differently to differentiate between them. This is illustrated in Figure 5. Using this technique, the user can simultaneously scale and position the figure in a manner which strongly mimics how similar transactions are performed in the everyday world.
Figure 5: Using
two hands to stretch a square.
ParticipantSubjects may reacquire the encompassing figure to make adjustments by dragging a handle with the right cursor as in the unimanual technique. The right hand is attached at the control point that was pressed and the left hand is attached to the control point at the opposite corner of the figure. The figure may then be manipulated using both hands.
Toolglass with Two Hands (T2)
This condition integrates the previously described two-handed stretching method with the Toolglass technique introduced by Bier et al. (1993). Tool selection is effected using a "click-through tool," using the right hand to press through a transparent tool palette whose location is controlled by the left hand. What is novel compared to previous click-through tools reported in the literature is that once the tool is selected, the tool palette disappears, and the "other" hand now controls the position of one corner of the encompassing figure instead of the position of the palette. The right hand controls the opposing corner. The users may then size and position their encompassing figure accordingly, the same as in the two-handed stretching technique. Once they release the input device, the palette reappears at their left hand. The participantsubject may reacquire and manipulate the figure in the same manner as the two-handed stretching technique.
In all three techniques the size of the palette menu was kept the same to normalize across experimental conditions.
The Hypotheses
Our analysis began with the recognition that this class of interaction is a compound task. The steps, as implemented in conventional one-handed technique (C1), are as follows:
Frequently, the user must repeat steps 2 to 4 of the aforementioned list since typically either the vertical or horizontal alignment of the first specified control point is incorrect (as also demonstrated previously in Figure 2). An analysis of this simplified characterization of the task is shown in Figure 6.
The first phrase stands for the selection of the tool. The onset of the second phrase represents the selection of the first control point (press), and the trailing edge of the phrase, the specification of the second control point (release). These two control points define the response’s position and size. The dotted additional phrases represent potential phrases that may be required to correct the placement of the stimulus corners to complete the task.
What Figure 6 also shows is at what stage in the process the user commits to the specification of key parts of the transaction. When the user initially presses down to begin drawing their selection, they have committed to the placement of that ‘half’ of the figure. That control point cannot be moved until the drag is released and the point is reacquired by pressing on the handle at that control point. From this, we can derive the likely cause of potential error, namely that the specification of the first control point must be made before there is any feedback from the transaction to aid in its proper alignment.
Recognizing the potential error and its cause (insufficient feedback at the time of commitment), there are two strategies to pursue. We could add the feedback at the time. This could be done with something like cursor cross hairs which extend to the edges of the screen. While such a strategy can be useful for the tasks concerned in this study, displaying large full screen cross hair cursor can be distracting in other tasks. An alternative is to defer commitment until the feedback exists. The two new techniques that we used in this study chose the second, more general strategy.
The "Stretching with Two Hands" (S2) technique’s phrasing is illustrated in Figure 7. Again there are two phrases. Tool selection is as with the C1 case, a single click. The difference lies in the fact that while sweeping the area encompassing the presented stimulus, the initial corner is not permanently "fixed" as it was in the C1 case. The user may simultaneously move the two defining control points of the figure, that which is attached to their left hand and that which is attached to their right hand.
Figure 7: Characteristics
of enclosing a target using the "Stretching with Two Hands" technique (S2).
What this means is that one is not committed until the end of the second phrase, when they release their drag. The user sees the results, where their encompassing figure lies, before committing to the final placement of either control point.
The "Toolglass with Two Hands" (T2) technique is characterized in Figure 8. In this case, selection of the tool and sweeping out of the area are integrated into a single phrase. After tool selection, the user may manipulate the encompassing figure with both hands, as in the S2 technique.
With the one-handed technique, the "cognitive load" of the visualization process varies with the size and shape of the encompassing figures. For example, in the case of encompassing an oval with a rectangle, the user must visualize where to initially press down to begin their sweep. This choice is made by mentally projecting lines from the edges of the oval and determining the point where they intersect (as demonstrated in Figure 9, the user must visualize initial point I). As the distances from point a to point b as well as from point c to point d increase (i.e. as the oval increases in size), it becomes harder and harder to reliably mentally visualize the extension lines and accurately pick point I. In other words, the larger the target figure, the higher the cognitive load of visualization load the unimanual technique imposes to the user.
The case where the user is required to encompass a rectangle with an oval is more cognitively demanding. As in typical graphical drawing packages, the oval is specified by sweeping out a rectangle into which the oval is drawn. Thus, in order to correctly encompass a rectangle with an oval the user must mentally visualize where the oval must be and then mentally visualize the box which defines that oval, which means visualizing extension lines from the edges of that oval (similar to the visualization discussed in Figure 9) to determine the initial press down point I (visualization graphically depicted in Figure 10). If this guess of point I is incorrect the user can still sweep out the oval, but in contrast to the previous example, the release at point C is not visually verifiable. This is due to the fact that subsequent adjustments of point I will affect the overall placement of the oval. Thus the user may be forced to travel repeated times between their initial point I and the completion point C in order to achieve criteria. Again, these effects will be enhanced by size.
With two handed input techniques, the user does not have to switch between the two control points and hence gain motor efficiency advantage. Furthermore, the need to mentally visualize the final locations of the lower than natural level of elements (control points) is also eliminated (externalized). The user hence also gains a cognitive advantage.
Therefore, working from these analyses, we established the following predictions.:
Based on the previous evidence of two-handed advantage (Buxton and Myers,
1986; Kabbash, Buxton and Sellen, 1994), albeit with a new class of tasks,
we predicted:
Hypothesis 1: Both two-handed techniques (S2 and T2) will outperform the conventional unimanual technique (C1).According to our notion of chunking and phrasing, which expresses how efficient interactions are often marked by ‘chunking’ task elements into greater units (Buxton, 1986). In the current task, the unimanual technique (C1) afforded the user to operate at the lowest level of chunking of the three techniques. Different components of the task (tool selection, sweeping, and multiple resizing) had to be carried out by multiple, separated motor tension "chunks" (press and release cycles, see Figure 6). The stretching with two hands (S2) technique, integrated the multiple resizing cycles into one continuous motor tension chunk, but still kept tool selection as a separate unit (Figure 7). The tool glass technique (T2) integrated the entire task, from tool selection to area sweeping into one motor tension cycle (Figure 8). We hence expected:
Hypothesis 2: Performance will improve with level of integration (chunking), from C1 to S2 and from S2 to T2.As complexity increases, it becomes ever harder to mentally position the encompassing figure’s corners. In the absence of feedback, it becomes increasingly difficult to visualize the task. Such a visualization process is cognitive, and that it will play a significant role in participants’ performance. Such an analysis predicted:
The Apparatus
The experiment was performed on a Macintosh Power PC 6100/60 AV model with a 17-inch colour monitor with 832 by 624 pixel resolution, using a Wacom UD-1212R tablet (Wacom, 1994) with "Multimode" capabilities (the tablet can sense two different devices simultaneously on its surface).
ParticipantSubjects held a Wacom 4-button ‘puck’ in their left hand and a ‘stylus’ in their right hand. For the one-handed technique, only the stylus was used. These two devices are shown in Figure 12, held on the tablet.
The stylus’ "hot-point" is its tip. To move the screen cursor with the stylus, users held the stylus like a pen and moved it slightly above the surface of the tablet (0-5 mm). If it was lifted too far from the surface of the tablet the cursor would freeze. It would resume movement once the device was brought close enough to be sensed again. Pressing down with the tip of the stylus was recognized as equivalent to a mouse button press.
The stylus and puck are untethered. For this experiment, none of the buttons on the puck were used, nor the barrel switch on the side of the stylus. The hot-point of the puck was also offset such that the screen cursor appeared to the right of the physical device to reduce physical collisions of the puck and stylus on the tablet surface.
The tablet was scaled such that a 21.7 centimeters (across) by 17.7 centimeters (down) area on its surface mapped to the complete computer monitor.
The ParticipantSubjects
Fifteen peoplesubjects (Ss) participated in this experiment, 12 males and 3 females. All participantsSs were right-handed. None had significant prior experience using an absolute positioning device such as the tablet, but most were very experienced computer users (use one every day). Most were novice users of computer drawing packages (use one 0-25% of their computer experience).
The Design and Procedure
Each participant subject performed all three techniques in a within-subjects design. The experiment was a 3 x 6 x 4 factorial experiment (Technique x Figure x Size). Technique order was counterbalanced using a Latin Square design: 5 participantsSs performed in the order of C1 S2 T2, 5 in S2 T2 C1 and 5 in T2, C1 S2.
ParticipantsSs were first instructed verbally, outlining the general requirements of the task. Before attempting each technique, they received verbal instructions outlining the properties of the technique and one practice block consisting of 12 trials with random shape, size and length to width ratio.
After practicing, participantsSs performed 2 blocks of trials. Each block consisted of 24 trials presented in a random order. The presented figure in each trial would be one of six shapes (Square, Rectangle, Circle, Oval, Equilateral Triangle, and Isosceles Triangle) and four discrete sizes (50, 130, 200 and 280 pixels). Each combination of size and shape would be seen once during the block totaling to the 24 trials in a block (6x4). The ratio of length to width for asymmetric shapes without rotational symmetry was constant for all at 1:2.
ParticipantsSs took approximately one hour to complete 2 blocks for each of the three techniques. After completing all three techniques participantsubjects were administered a questionnaire to elicit subjective feedback.
The Results
Total trial time was calculated as the time from selection of the tool to completion of the task. Overall, total trial time differed very significantly amongst techniques (F2,24 = 28.22, p < 0.0001). Table 1 shows the mean total trial time for the three techniques. We can see that two-handed techniques on the average save about 17% of the time it takes to do the same task using the conventional one handed technique (C1).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Table 1: Mean total trial completion time for Experiment 1.
Pair-wise contrast tests for differences between means showed that
both two-handed techniques were significantly better than C1 (p < 0.0001),
but that the two two-handed techniques (S2 and T2) were not significantly
different from each other (p = 0.16).
Subjectively, when asked which technique they liked best, users preferred S2 (stretching with two hands) over T2 (Toolglass with two hands), 8 to 6 with one user preferring C1. When questioned about which technique they liked the least, 10 users expressed dislike for C1, and 5 users expressed dislike of T2. None of the users chose S2 as their least liked technique.
Measurements of the left and right locations were recorded every third of a second which enabled us to calculate rough speed estimations for the left and right hands. The speeds of the left hand tended to be slower than the speeds of the right. A sample is shown in Figure 13 from a participantsubject that accomplished the task in one attempt (no need to reacquire the control points) using the S2 technique. A sample of one-handed trial (the C1 technique) is shown in Figure 14 from a participantsubject who took four attempts to achieve task completion (had to reacquire four control points before getting the encompassing figure correct).
Thus far, the results have indicted that two-handed techniques were superior overto the one-handed technique for practical purposes. Theoretically, the performance difference could be a result of either physical efficiency or cognitive characteristics of the techniques. One obvious physical motion difference between one-handed and two-handed techniques lies in the (re)acquisition of the control points (handles). With the one-handed technique, the user has to switch between two control handles (Figure 14). Each reacquisition of a control handle is a Fitts’ law task. This alone could account for the entire performance difference between the one-handed vs. the two-handed techniques. It is therefore informative to remove the control handle reacquisition times from all the conditions, although it is an inherent part of one-handed manipulation.
After removing the control point reacquisition parts of the task, the one-handed and two-handed techniques became equivalent in terms of amount of motion needed to complete the task. The cause of performance difference between one-handed and two-handed after the removal could still be twofold: the physical motor efficiency difference due to the parallelism in the two-handed conditions and the cognitive visualization difference between the one-handed and the two-handed conditions. However, as analyzed earlier, two factors, size and shape of the targets, changed the amount of mental visualization needed. They therefore could serve as indicators of the importance of the cognitive factors in the performance differences between conditions.
The results after the removal of control point reacquisition times are shown in Table 2. After the removal, we see no significant mean performance difference between the one-handed technique and two-handed techniques. However, there was a significant interaction between technique and size (F6,24 = 3.34, p < 0.01).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Table 2:Mean total trial completion time after removal of reacquisition time for Experiment 1.
As shown in Figure 15b (performance time after reacquisition time was removed), for the largest target whose final correct control points were the most difficult to visualize, two-handed techniques were significantly better than one-handed (p < 0.05 for C1 vs. S2 and p < 0.05 for C1 vs. T2). For the smallest target that did not require much planning and visualization, two-handed was in fact inferior to one-handed (p < 0.001 C1 vs. S2) or was not statistically different from the one-handed technique (p = 0.84 C1 vs. T2). If the advantage of the two-handed conditions were all due to the parallelism in the physical actions, we would expect at least equivalent performance to the one-handed for the smallest targets. Clearly the cognitive factor is an important contributing component to the performance difference between the one and two-handed techniques.
(b)
(b)
As cautioned by some experimental psychologists (Poulton, 1966; Poulton, 1989), a common pitfall of within-participantsubjects design of an experiment is the asymmetrical skill transfer effect. That is, with a within-participantsubjects design, the actual skill transfer from one condition to another might not be symmetrical, even though participantsubjects’ exposures to the conditions are equalized. This appears to be true of this experiment. The interaction of technique and order was shown to be a very significant factor (F4, 24 = 11.65, p < 0.0001).
The ordering effect did not significantly influence the trend that as size increased, the two-handed techniques outperformed the one-handed technique to a greater extent. The three way interaction of Technique x Size x Order was insignificant both before (F12, 72 = 0.77, p = 0.68) and after (F12, 72 = 0.79, p = 0.66) the control points reacquisition time removal. Likewise, Technique x Shape x Order was also insignificant both before (F20,120 = 1.32, p = 0.18) and after (F20,120 = 1.33, p = 0.17) the control points reacquisition time removal. This shows that the evidence of cognitive benefits in the two-handed conditions still hold regardless of the asymmetrical transfer effect.
We believe the asymmetrical skill transfer is related to the fact that the users did not have enough time to learn how to use the tablet (recall none had significant prior experience). This called for a follow-up experiment that would give users more time to learn the input devices as well as the techniques. In addition, following the evidence of cognitive benefits in two-handed interaction, we decided to use only the cognitively more demanding targets (large sized square and triangle shapes). We also changed the feedback design so as to give more rapid feedback as discussed in the following description of Experiment 2’s Task. It was observed during the first experiment that users frequently missed the target figure by a pixel or two when they thought they were accurate. The user would not know they missed the target until they released the input device. This delayed feedback might hinder one of the cognitive advantages of two-handed input (rapid feedback towards goal state).
Experiment 2
A follow-up experiment was conduct mainly for the purpose of clarifying if asymmetrical skill would still be a significant factor if more practice was given to each technique tested. In addition, the feedback method that might have hindered the task performance was also modified.
The Task
As mentioned, more rapid feedback was provided in this experiment. This was done such that when the user’s encompassing figure arrived at the correct spot, the red highlights would appear, regardless of whether the drag had been released. Once the drag was released, the figure was evaluated again and if all points still matched the system would record the trial as complete.
The Techniques
In the first experiment there was no significant difference between the performance of the two bimanual techniques. We chose to study only one of the two bimanual techniques, the "Stretching with Two Hands" (S2) technique. This technique was chosen over the Toolglass technique (T2) because no further improvement was found with the T2 technique, which integrates menu selection into two handed input. We also noted in the first experiment that the T2 technique was disliked by most participants.
The Apparatus
We used the identical experimental setup as for the first experiment.
The ParticipantSubjects
Eight peoplesubjects participated in this experiment, 6 males and 2 females. All participantsubjects were right handed and had not participated in any previous two-handed experiment. Only one participantsubject had significant prior experience using a tablet. All participantsubjects were experienced computer users (used one every day).
The Design and Procedure
We restricted this experiment to the shapes and sizes which are previously analyzed as cognitively more demanding. Only the square, rectangle, equilateral triangle and isosceles triangle were used for presentation shapes. Therefore the encompassing figure was limited to an oval shape. ParticipantSubjects were still required to choose the oval tool to start each trial. A random set of sizes, +/- 30 pixels of the largest size used in the first experiment (280 pixels), was generated and presented to all participantsubjects. The ratio for asymmetric shapes without rotational symmetry was constant for all at 1:2.
The overall reduction of experimental conditions (the number of techniques as well as reduced set of stimuli) allowed us to give more time for participantsubjects to perform each technique with each type of target. In addition, we also reduced our block size to 12 trials per block.
ParticipantSubjects were first instructed verbally, outlining the general requirements of the task. Before attempting each technique, participantsubjects received verbal instructions outlining the properties of the technique and completed one practice block consisting of 12 trials with random shape, size and length to width ratio.
All participantsubjects were able to complete 5 blocks of each technique in less than one hour. ParticipantSubjects were administered a questionnaire after completion of all trials to elicit subjective feedback.
The Hypotheses
The order effect probably confounded the main effect (performance difference between techniques) in the first experiment. Such an order effect might reduce to be insignificant, as more experience was given to each technique in this experiment. The selection of the size and the shape was intended to be more cognitively demanding for unimanual techniques and the design of instantaneous feedback of being on target was intended to facilitate the visualization process (another cognitive factor). With these changes, we hypothesized that the two-handed condition would outperform the one-handed condition to a possibly greater extent in this experiment than in Experiment 1.
Results
Total trial completion time differed significantly between the two techniques
(F1,6 = 89.08, p < 0.0001). Table 3 shows the mean total
trial time for each technique. The two-handed technique resulted in time
savings of about 39%.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Table 3: Mean total trial completion times for Experiment 2.
In comparison to the results from the first experiment, the mean completion time using the one-handed technique has increased about one second for this experiment while the mean completion time using two-handed technique total time has decreased by over two seconds. Recall that in this second experiment we used only the more cognitively demanding targets (larger sizes and difficult shapes). We also provided rapid feedback and more practice with both techniques.
The interaction of technique and order is not significant (F1,6 = 9.64, p = 0.77), suggesting the asymmetrical skill transfer is no longer a significant effect. We therefore conclude that the differenent characteristicsce between the two techniques is stronger than the inherent instead of a by-product of the skill transfer transfer effect.
Learning significantly improved both techniques (p < 0.005) and the two-handed technique (S2) was consistently better than C1 for all blocks (Figure 17). The interaction of technique and block was not significant (p = 0.27). We hence do not expect further practice will change the results of this experiment.
Again we removed the control point reacquisition time from the total completion times in order to remove the time-motion disadvantage of the one-handed technique. Technique was still significant (F1,6 = 25.46, p < 0.005). Figure 18b shows the results of our analysis in relation to the shape of the targets. For all shaped targets, S2 was faster than C1 (p < 0.01).
(a)
(b)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Table 4: Mean total trial completion times with reacquisition times removed for Experiment 2.
We also recorded the number of errors, i.e. the number of times that the participantsubject had to re-acquire control point(s) and to adjust the encompassing figure. Variance analysis showed that the only significant factor to error rate was technique (p <0.001). None of the other main effects or interaction terms was significant. On average participantsubjects made 4.2 adjustments with C1 technique and only 0.77 adjustments with the S2 technique in each trial of the task. Figure 19 and 20 illustrate the number of errors as a function experimental blocks and shape. The large error disparity between the two techniques was in conformity with our early analysis: With the one handed technique, due to the cognitive difficulty of mentally visualizing the control points forced upon the user, the participantsubjects often had to make a commitment to a control point that was not accurate so they had to re-acquire and adjust it later. In contrast, with the two handed technique, the need of searching for the control point became externalized. The user only had to rely on perceptual and motor skills to explore the accurate control points with their resulting consequence continuously displayed. Interestingly, even with the two handed technique, the participantsubjects still occasionally dropped the control points and then realized they were not on target.
Different from trial completion time, which was improved significantly over the 5 blocks of the experiment, the number of errors did not change significantly in the course of the experiment (p = 0.74, Figure 19), suggesting that the cognitive difficulty of projecting correct control points and the misperception of being on target were too inherent to overcome by learning in the experiment.
The two experiments presented in this paper provide rich information on how two-handed input differs from one-handed input. This section organizes our major conclusions around our initial hypotheses.
Another factor that might reduce the effectiveness of the Toolglass technique is that even though an offset was built into the system, by offsetting the effective input location of the puck, the participantsubjects still expected the puck and stylus to physically overlap because on the screen the palette and the cursor overlap. This confusion could arise due to the fact the input space was shared by two hands, both stylus and puck operated within the same coordinate space. In fact, one participantsubject initially lifted the pen and attempted to click within the plastic cross-hairs area of the puck.
In summary, the present work extends two-handed input into a new class of tasks. Furthermore, the experiments and analyses advance the theoretical understanding of two-handed input. As our literature review showed, two-handed input has made significant progress in the past decade. The introduction of the kinematic chain theory of human bimanual function greatly enriched the theoretical foundation of two-handed input design, in terms of the different functions each hand has. However, the KC theory does not explain why two hands can be better than one hand for computer input. The present work provides a complimentary perspective. We base our analysis of two-handed input on the concept of chunking and phrasing and argue that the advantage of two-handed input is both manual (physical) and cognitive. When a user can manipulate the entire object with two hands as an integrated chunk that is greater than the separate elements as in unimanual input, both manual (elimination of repeated control points reacquisition) and cognitive (reduction of mentally visualizing the control points) advantages were gained. Our experiments showed that as the task became more cognitively demanding (more visualization required), the advantage of two-handed input over one-handed input became more pronounced. We believe that to give users the ability to use both hands and manipulate the task at a natural level of chunking is a logical and an inevitable trend for a broad range of human-computer interaction applications.
Acknowledgement
We thank the anonymous reviewers and our editor Bonnie John for their helpful and constructive reviews which has substantially improved the paper.
References
Anderson, J.R. (1982). Acquisition of Cognitive Skill, Psychological Review, 89( 4), 369-406.
Bier, E.A., Stone, M.C., Pier, K., Buxton, W. and DeRose, T.D. (1993). Toolglass and magic lenses: The see-through interface. In Proceedings of SIGGRAPH ‘93, Annual Conference Series (Anaheim, California), Computer Graphics 27, 73-80.
Buxton, W. (1986). Chunking and phrasing and the design of human-computer dialogues. In Proceedings of the IFIP World Computer Congress (Dublin, Ireland), 475-480. Reprinted in Baecker, R., Grudin, J., Buxton, W. and Greenberg, S. (Eds.). (1995). Readings in Human-Computer Interaction: Towards the Year 2000. San Francisco, CA: Morgan Kaufmann, Inc., 494-499.
Buxton, W. and Myers, B.A. (1986). A study in two-handed input. In Proceedings of CHI ‘86 Conference on Human Factors in Computing Systems (Boston, Massachusetts), 321-326.
Card, S., Moran, T. & Newell, A. (1980). The Keystroke Level Model for User Performance Time with Interactive Systems, Communications of the ACM, 23(7), 396-410.
Chatty, S. (1994). Extending a graphical toolkit for two-handed interaction. In Proceedings of the ACM Symposium on User Interface Software and Technology (Marina del Ray, California), 195-204.
Dillon, R.F., Edey, J.D. and Tombaugh, J.W. (1990). Measuring the true cost of command selection: Techniques and results. In Proceedings of CHI ‘90 Conference on Human Factors in Computing Systems (Seattle, Washington), 19-25.
Fitts, P.M. (1954). The information capacity of the human motor system controlling the amplitude of movement. Journal of Experimental Psychology 47, 6, 381-391.
Fitts, P. M., & Seeger, C. M. (1953). S-R compatibility: spatial characteristics of stimulus and response codes. Journal of Experimental Psychology, 46, 199-210.
Flowers, K. (1975). Handedness and controlled movement. British Journal of Psychology, 66 (1), 39-52.
Guiard, Y. (1987). Asymmetric division of labor in human skilled bimanual action: The kinematic chain as a model. Journal of Motor Behavior 19, 4, 486-517.
Guiard, Y. and Ferrand, T. (1996in press). Asymmetry in bimanual skills. In Elliott, D. (Ed.) Manual Asymmetries in Motor Performance. Boca Raton, FL: CRC Press, 175-195.
Hinckley, K., Pausch, R., Goble, J.C. and Kassell, N.F. (1994). Passive real-world interface props for neurosurgical visualization. In Proceedings of CHI ‘94 Conference on Human Factors in Computing Systems (Boston, Massachusetts), 452-458.
Kabbash, P., Buxton, W. and Sellen, A. (1994). Two-handed input in a compound task. In Proceedings of CHI ‘94 Conference on Human Factors in Computing Systems (Boston, Massachusetts), 417-423.
Kirsh, D. and Maglio, P. (1994). On distinguishing epistemic from pragmatic action. Cognitive Science 18, 4, 513-549.
Krueger, M. (1983). Artificial Reality. Reading, MA: Addison-Wesley.
Owen, R., Kurtenbach, G., Fitzmaurice, G., Baudel, T. and Buxton, W. (1996). Bimanual manipulation in curve-editing tasks. Unpublished Manuscript.
Peters, M., Constraints in the performance of bimanual tasks and their expression in unskilled and skill subjects, The Quarterly Journal of Experimental Psychology (1985) 37A, 171-196.
Poulton, E.C. (1966). Unwanted asymmetrical transfer effects with balanced experimental designs. Psychological Bulletin 66, 1, 1-8.
Poulton, E.C. (1989). Bias in Quantifying Judgments. Hillsdale, NJ: Lawrence Erlbaum Associates.
Rosenbloom, P.S. (1985). The Chunking of Goal Hierarchies: a Model of Practice and Stimulus-Response Compatibility, Doctoral Dissertation, Dept. of Computer Science, Carnegie Mellon University.
Sachs, E., Roberts, A., and Stoops, D. 3-Draw: a tool for designing 3D shapes. IEEE Computer Graphics and Applications 11, 6 (Nov. 1991), 18-26.
Schmidt, R.A., Zelaznik, H., Hawkins, B. Frank, J.S. and Quinn, J.T. (1979). Motor-output variability: A theory for the accuracy of rapid motor acts. Psychological Review 86, 5, 415-451.
Shneiderman, B. (1983). Direct manipulation: A step beyond programming languages. IEEE Computer 16, 8, 57-69.
Simon, H. (1974). How big is a chunk? Science, 183, 482-488.
Smith, D.C., Irby, C., Kimball, R., Verplank, W. and Harslem, E. (1982). Designing the Star user interface. Byte 7, 4, 242–282.
Summers, J.J., T.M. Kennedy, T.M., Strategies in the production of a 5:3 polyrhythm, Human Movement Science 11 (1992) 101-112
Sutherland, I.E. (1963). Sketchpad: A man-machine graphical communication system. AFIPS Conference Proceedings, 23, 329-346.
Todor, J.I. and Doane, T. (1978). Handedness and hemispheric asymmetry in the control of movements. Journal of Motor Behavior 10, 295-300.
Wacom Corporation, Ltd. (1994). Wacom UD-1212R, UD-1218R/RE and UD-1825R Serial Tablet User’s Manual.