Hardock, G., Kurtenbach, G., and Buxton, W. (1993). A Marking Based Interface for Collaborative Writing. Proccedings of UIST'93., 259-266.


A Marking Based Interface For Collaborative Writing

Gary Hardock, Gordon Kurtenbach and William Buxton

Department of Computer Science
University of Toronto
Toronto, Ontario
Canada  M5S 1A4


We describe a system to support a particular model of document creation.  In this model, the document flows from the primary author to one or more collaborators.  They annotate it, then return it to the author who makes the final changes.  Annotations are made using conventional marks, typically using a stylus.  The intent is to match the flow of paper documents that we observed in the everyday world, and how they are marked up. The system is very much modeled on Wang Freestyle (Perkins, Blatt, Workman and Ehrlich, 1989; Francik and Akagi, 1989; & Levine and Ehrlich, in press).  Our contribution is to incorporate mark recognition into the system and to explore some novel navigation tools that are enabled by the higher-level data structures that we use.  The system is described and the results of our limited user-testing reported.


Collaborative writing occurs in many different forms (Posner 1991).  We describe a system to support a particular model of document creation.  In this model, the document flows from the primary author to one or more collaborators.  They annotate it, then return it to the author who makes the final changes.  Annotations are made using conventional marks, typically using a stylus.  This flow of the document is illustrated in Figure 1.

The intent is to match the flow of paper documents that we observed in the everyday world, and how they are marked up. The system is very much modeled on Wang Freestyle (Perkins, Blatt, Workman and Ehrlich, 1989; Francik and Akagi, 1989; & Levine and Ehrlich, in press).  With Freestyle, annotations are made with a stylus, very much as would be made with paper and pencil and by voice.  In what follows, we are only concerned with stylus annotations.

With Freestyle,  the copy of the document that is distributed is only a "dumb"  snap-shot of the original.  Annotations are integrated as a separate layer to the "snap-shot." There is no recognition of the markings.  To edit the final document, then, the user needs two versions:  the "dumb" annotated copy and the original.

Our contribution is to incorporate mark recognition into the system and to explore some novel navigation tools that are enabled by the higher-level data structures that we use.  In what follows, we describe the system and the results of our limited user-testing.

Figure 1:  The Asynchronous Collaborative Writing

The principal author creates a document and then sends copies of it to collaborators.  The collaborators then annotate their copies and send them back to the principal author.  The principal author reviews the suggestions and possibly incorporates them in a revised version.

MATE:  Markup Annotator Text / Editor

Documents can be prepared with any word processor, including MATE, which can function as a marking-based editor.  (As a prototype, however, it is limited in functionality.  It only supports delete, move and insert commands, as well as some novel navigation functions.  Text is entered by the keyboard.)  Figure 2 shows an example of MATE in “edit mode”.

(a)                                                                                 (b)
Figure 2:  MATE in “Edit Mode”.
(a) A user draws a move command.  (b) After lifting the pen, the editing command is performed.

Copies of this document can then be emailed to the collaborators.  Each reviews the their copy using MATE in “annotation mode”.  In this mode a collaborator can specify suggested changes by marking up the document.  As with paper documents and Freestyle,  markings are not interpreted as commands.  They are treated strictly as annotations.  These marks do not necessarily have to be editing commands.  For example, a marking could be something as vague as “reword this paragraph”.  An example of various annotations is illustrated in Figure 3.

Figure 3:  MATE in “Annotation Mode”.
In annotation mode, users mark up a document in MATE just as if they were marking up a paper document.

Marked up documents are returned to the primary author, who does the actual revision.  This is done using two views of the document (Figure 4).  The left view shows the marked up document.  Each reviewr's annotations appear in a different colour.  Additional annotations can be made in this window, but the underlying text will not change.  The right view shows the current version of the document.  No annotations are visible since any marks made in this window are interpreted as commands and executed immediately.   This view is similar to the stand-aloneeditor window than may have been used in creating the original document.

Figure 4:  MATE in “Incorporation Mode”.

In incorporation mode, a user can view the annotated document, and select which annotations to incorporate. Annotations t(seen in the left window) that have been "executed" appear as thin lines (eg., "ed").  Annotations that have not been executed appear as thick lines.  Annotations are colour coded according to who made them.  Annotations that represent commands can be executed by selecting them with the stylus.  Annotations that have been executed can be "undone" by selecting them.  The current state of the document appears in the right hand window.  The user can navigate (scroll) independently or synchronously in each window.

The two views are iwork in concert, however.  A user can point to an annotation in the left view and ask the system to perform it.  The resulting changed document appears in the right view.  The important characteristics of this design are that:

Other reasons for this split view design are identified later when the specific design issues are discussed.

Combining Annotations and Edit Commands

Marks have already been used extensively in annotating documents (Chow and Kim 1989; Go 1991, Welbourn and Whitrow 1988; Wolf, Rhyne and Ellozy 1989).  They are also being used to specify commands to computer applications.  There are several properties of marks which make them good for each purpose: All of the above properties make marks well suited for annotating documents.  The fact that most people can make and understand marks gives a compelling reason to build mark-based interfaces to computer applications.  But it is because marks possess all of these properties that enables the same marks to be used as both an annotation and as an editing command.

One way of comparing the annotation process and editing a document on a computer is that in the first case a person is communicating with another person, whereas in the second case a person is communicating with a computer.  The goal then becomes to design a method in which a person can communicate to both another person and to a computer application.

The fact that marking commands to the computer application are visible is the key.  If, instead of immediately interpreting a marking command, the computer does not process the mark but simply leaves the mark visible, the mark can be thought of as an annotation.  From this point of view, annotations are deferred edit commands.  In terms of the different modes of MATE, annotation mode can also be called deferred mode, and edit mode is immediate mode.

Functionality and Capabilities of MATE

In addition to the benefits and issues concerning the use of markings for editing and annotating, there are many advantages and issues when both uses are combined in an integrated system.  In this section we describe some of these.

Do by selection:

As mentioned earlier we can select markings in the annotation view to be performed in the edit view.  The selection is accomplished by tapping on the marking.  Feedback is provided by making the marking thinner, indicating that it has been performed.  Note that this selection is possible because both the user and the computer can understand the editing command identified by the marking.  Figure 4 shows the result of selecting the insert and delete annotation marks for incorporation.

History Mechanism

The feedback provided by thinning the markings which have been performed provides a history of the annotations which have been incorporated.  However, this graphical history mechanism is spatial in nature, not chronological.  This is much more useful as the order of incorporating annotations is unimportant, whereas the locations of the annotations immediately tell us what text the commands have been applied to.  For example, in figure 4, we immediately notice that the insert and delete annotations have been performed and to what pieces of text they were applied to, but we do not know when or in what order.

Undo by selection

Just as a mark can be incorporated by tapping on it, a mark can also be unincorporated by tapping on it again.  This is only possible because the annotation marks are always visible, a result of having the two views of the document.  Note that Undo by selection is not order dependent.  Figure 5 shows an example of how this works.  It does not matter whether the delete or move command was performed first, either command can be directly undone.  Note that the annotations correspond to a specific piece of text, not the position of the annotation.  Such a mechanism for undoing text-editing commands has never been built before.


By combining Do and Undo by selection we have a method for previewing the results of suggested annotations.  If a user wishes to preview what would happen if they incorporated an annotation, they can Do it, examine the results, and then Undo it.  Note that because the Undo is non-chronological any combination of annotations can be previewed.  This allows the user to try “what if” scenarios, with only the cost of tapping on the annotations.  Note that the user need not be the principal author.  It could be used by a collaborator wanting to see the results if the annotations were incorporated.

(a ) 



Figure 5:  Undoing irrespective of temporal order.

The initial text is shown in (a).  (b) is the result after the deletion is performed.  (c) is the result after the move is performed.  After undoing the delete command, the desired result is (d).

Enhancement of Understanding

The ability of the system to understand the meaning of a mark lessens the possibility of a misunderstanding between the annotation’s writer and reader. Table 1 shows that the addition of computer recognizing commands can aid in the communication between the annotation’s creator and reader.
Understanding Computer
Reader Understands MisInterprets Does not Understand
Understands mutual understanding user will ignore computer’s misinterpretation computer neither aids nor hinders user interpretation
MisInterprets Computer understands, may help the reader understand the marking  user and computer may have different misinterpretations, which may cause the user to possibly try a different interpretation
Does not Understand User may accept the computer’s misinterpretation

   Table 1:  Enhancing the Understanding between the creator and reader of an annotation.

If the reader does not understand or misinterprets the meaning of a marking, the computer may be able to understand it

Viewing the Annotations of Several Authors

The annotations of several collaborators can be overlaid in the annotation view.  To differentiate among the various collaborators markings, each set is displayed in a different color.  Note that this is possible because the markings are separate from the text document, so different characteristics can be added to each set of markings.  This is analogous to writing on a transparency overlaid on top of a paper document and writing on the transparency.  Then several marked up transparencies can be placed over the paper document.

A preliminary study using transparencies was conducted to identify the issues and advantages of overlaying the annotations.  The results indicated that the usefulness of this feature depends upon the density of annotations on the page.  In some cases a single transparency was cluttered, and overlaying several transparencies made the annotations illegible.

As the problems of clutter are also found in a single set of annotations, we decided to address the more general problem of reducing the density of the markings.  This is accomplished by providing support functions such as “Hide Set of Marks”, “Show Set of Marks”, “Hide Mark”, and “Show Hidden Marks”.

Broken Move, Multiple Buffers and Placeholders

As the editing marks also serve as annotations, we have examined the markings used in the everyday marking-up of paper documents to gain insights into the design of the editing commands.  One command we have “borrowed” from pen and paper is the broken move command, shown in Figure 6.  Instead using a “star’ symbol to move to and from, we use a marking menu containing various symbols termed placeholders.


Figure 6:  Moving text across pages using “move to star” and “move from star”.

Another way of thinking about this command, besides as a broken move, is as two commands, move to buffer with ID, and move from buffer with ID.  This allows multiple text buffers, each with a unique placeholder symbol to identify it.  It is important to note that the user chooses which placeholder to move text into, thus providing a strong connection between the placeholder and the text it contains.  Also, because the placeholders containing text are visible, it is trivial to determine what text is contained in a placeholder buffer.

The Supporting User Interface

The overall design of the user interface allowed the system to provide the functionality mentioned above, but there are several additional interface components which are needed to support this functionality

Interacting with the Markings

We mentioned above that markings could be done and undone by tapping on them.  But other operations on the marks are also necessary, for example, the need to hide or erase marks.  In order to provide this functionality marking menus (Kurtenbach & Buxton, 1991) were implemented.  This gives a logical extension to tapping on the mark.  If the user taps on the mark, it is done or undone. If, however, the user makes a mark starting from an existing mark it is interpreted as a marking menu selection.  Figure 7 shows the marking menu for operations on marks.  GoTo and GoBack are discussed in the section on coordinating the two views.

Figure 7: Marking menu for operations on annotation markings.


Instead of using scrollbars, our approach to navigation was based upon the handling of a piece of paper.  By holding down the middle button of the mouse or the button on the pen, the user enters navigation mode.  In this mode, the “page” or window of the annotation or edit view can be grabbed and moved.  An upwards motion moves the document forward, which corresponds to moving the scrollbar or viewing window down.  If the motion is slow the document moves as if it has been grabbed and is being pushed.  If the motion is faster then the document moves as if the user had flicked a page forward or backward.

Navigation mode is actually implemented as a modified marking menu, shown in figure 8.  Page flicks are always too fast for the menu to pop up, but the page push commands can also be selected by the menu.  In addition to the page push and page flick commands, there are the GoTo and GoBack commands which are discussed in more detail in the next section.

Figure 8: Marking menu for navigation.

Coordinating the Annotation and Edit Views

Several problems appear due to the fact that there are two views of the same document.  Some of these are:

Linked Navigation

As the document changes, the annotation and edit views become more and more different.  Therefore it is not always clear how each view should react when a navigation command is performed in the other view.  One solution is to “link” the two views so that they move equal amounts.  This is accomplished by modifying the page flicks and page pushes.  If the user wants a linked movement, he or she first makes appropriate move command and then without lifting the pen, draws a line to the left or right.

GoTo and GoBack

A solution to all of these issues are the GoTo and GoBack commands.  There are two slightly different types of GoTo commands.  When GoTo Text is applied to a piece of text in one of the two views, the corresponding piece of text is found in the other view, and then the other view is aligned with the first view.  The piece of text is also highlighted.

GoTo Annotation is similar to GoTo Text, except that it is applied only to annotations.  This means that GoTo Annotation can only be used in the annotation view.  When GoTo Annotation is applied to a mark, the mark is interpreted, and if the interpretation is successful, the Edit view is aligned with the annotation view and the text affected by the command is highlighted.

Interface Alternatives

The general concepts discussed in the introductory sections could possibly be applied using interfaces other than those based upon the pen and paper metaphor, but the property that marks are visible and the capability to use the same mark as both an annotation and for specifying the editing command gives the marking-based approach definite advantages.

GUI Interfaces

For example, a GUI type of interface with a mouse and keyboard could be used to enter comments and specify commands.  This was partially achieved with the Collaborative Annotator (Koserak et.al 199?).  However it only allows annotations to be made, it is not a text editor.  The main problem with GUI type or direct-manipulation interfaces is that there is a separation between specifying editing commands and specifying annotations.  The actions used in specifying editing commands, such as menu selections, mouse movements, and button clicks do not leave a “visible audit trail” (Kurtenbach 19??), and therefore cannot be easily used as annotations.

Speech Interfaces

Speech is a very good means of communication among people, therefore it makes sense to use speech as a method of annotating a document.  But speech is not visible and is poor in specifying locations.  Also, speech is good for certain types of annotations, but poor for others.  In fact, speech is best for general comments and “wordy” comments, exactly the types of annotations for which mark-up annotations are poor for.  Therefore the ideal system would use a combination mark-based speech based interface.  However, as the speech based-interface is dependent upon the mark-based interface, we decided to first see would could be gained with a purely mark-based approach.  A hybrid system is left as future work.

Alternatives to the two views

Several problems occurred as a result of having two views of the document, the annotation and edit views.  There are alternatives, each of which has its own advantages and disadvantages.

One view for each set of annotations:

This is a good solution for the multiple sets of annotations problem.  But it magnifies the problems in coordinating a single annotation view with a single edit view.  Although the benefits of multiple annotation views may outweigh the disadvantages caused by these problems, we decided to concentrate on the issues concerning the coordination of one annotation and one edit view first and leave multiple annotation views as future work.

Single View:

Another alternative is to have a single view.  Although this might seem to solve the coordination problem, it only transforms it into another related problem.  As the document in the single view changes, the annotations will have to be modified to adapt to fit the current version of the document.  This is possible for annotations which can be interpreted as editing commands, but for general comments there is no way for the system to know how they should be transformed.  this is illustrated by the example shown in Figure 9.


Figure 9:  Attempting to Update a Single View.

(a) shows the original view, before the update.  (b) shows the result of an attempted update.  Such an update is very difficult as almost all of the marks need to be changed or moved, including marks strictly meant to be treated as comments.  Note that the placement of “Poor” is unknown.

These arguments do not mean that the two view approach is necessarily better.  It is an alternative with its own issues and problems.  It would be useful in the future to design the alternative interfaces and compare the advantages and disadvantages to each approach.

Experiences with MATE

MATE is still at the preliminary design / implementation stage but some limited user testing has been done.  The main purpose of the user testing was to determine if the underlying concepts were valid, and to identify any major problems with the system.  Also as the navigation command set was fully functional, it was tested in more detail.

Five users were placed in a mock scenario, in which they were to pretend that they were the principal author given a marked-up document to edit.  What they chose to incorporate was entirely up to them.

The results of the study showed that users choose to incorporate annotations by selection rather than by manually doing the edit themselves.  Issues concerning histories, undo, and previewing were inconclusive as the users did not use these features.

The task was not complicated enough to bring out the issues regarding the coordination of the two views.  However, there was some confusion between the GoTo Text and the GoTo Annotation commands.

The navigation testing was confounded by the stylus which was unreliable.  In particular this made page flicks very difficult to specify.  Several users preferred to use the scrollbar, instead of page flicks.  Another problem was the response time of the system for page movements, which often confused users about whether it understood their command or not.  These results do not conclude that page flicks do not work, but that more fine-tuning will be needed if they are to be useful and usable.

Summary and Conclusions

MATE was designed and built as one solution to the asynchronous collaborative writing problem.  As mentioned above, there are several alternative approaches and many directions for future work.  MATE is intended to be a first step in identifying, and addressing some of the problems with asynchronous collaborative writing.  Also several new issues and uses of marking-based interfaces have been developed in the process.


We would like to thank the members of the Input Research Group at the University of Toronto who provided the forum within which this work was undertaken.  Primary support has come from Digital Equipment Corp., Xerox PARC, and the Natural Sciences and Engineering Research Council of Canada.  Additional support has been provided by Apple Computer Inc., IBM Canada's Laboratory Centre for Advanced Studies, and the Arnott Design Group of Toronto.  This support is gratefully acknowledged.


Chow, D  &  Kim, J.  (1989).  Paper-Like Interface for Educational Applications, National Educational Computing Conference ‘89, Boston, Massachusetts,  337 - 344.

Francik, E. & Akagi, K. (1989).  Designing a computer pencil and tablet for handwriting.  Proceedings of the Human Factors Society 33rd Annual Meeting, 445-449.

GO corp. (1991a).  The Point of the Pen.  Byte, February, 211 - 221.

GO corp. (1991b).  Video Presentation at Stanford University, by Robert M. Carr, February 13.

Hardock, G.  (1991)  Design Issues for Line Driven Text Editing/Annotation Systems. Proceedings of Graphics Interface ‘91,  77 - 84.

Kosarek, J. L., Lindstrom, T. L., Ensor, J. R., & Ahuja, S. R.  (1990). A Multi-User Document Review Tool. In S. Gibbs, & A. A. Verrijn-Stuart (Ed.), Proceedings of Multi-User Interfaces and Applications . North-Holland: Elsevier Science Publishers, 207 - 215.

Kurtenbach, G.  &  Buxton, W.  (1991). Marking Menus  Proceedings of the ACM Symposium on User Interface Software and TechnologyUIST'91 .  137 - 144.

Levine, S.R. & Ehrlich, S.F. (in press).  The Freestyle system:  a design perspective.  In A. Klinger (Ed.).  Human-Machine Interactive Systems.  New York:  Plenum Press.

Perkins, R., Blatt, L., Workman, D. & Ehrlich, S. (1989).  Iterative tutorial design in the product development cycle.  Proceedings of the Human Factors Society 33rd Annual Meeting, 268-272.

Posner, I.R.  (1991).  A Study of Collaborative Writing.  Master’s Thesis,  University of Toronto.

Welbourn, L.K.  &  Whitrow, R.J.  (1988).  A Gesture Based Text Editor,  in D. Jones & R. Winder (Eds.), People and Computers IV, Proceedings of the Fourth Conference of the British Computer Society Human-Computer Interaction Specialist Group. Cambridge, Cambridge University Press,  363 - 371.

Wolf, C.G.,  Rhyne, J.R.  &  Ellozy, H.A.  (1989).  The Paper-Like Interface, IBM Technical Report RC 14615 (64399) 2/3/89, also in  G. Salvendy & M.J. Smith (Eds.), Designing on Using Human Computer Interfaces and Knowledge-Based Systems, Elsevier Science Publ, Amsterdam, 1989, 494 - 501.