Human Input to Computer Systems:  Theories, Techniques and Technology

William Buxton


Mark Billinghurst ,
Yves Guiard,
Abigail Sellen &
Shumin Zhai

1994, 2011 William Buxton

Some Context

In 2007 I published a book, Sketching User Experiences:  Getting the Design Right and the Right Design. The objective of that book was to emphasize the importance of bringing traditional design skills to interaction design, or the design of user experience.  Part of the motivating factor was that while the skills that were typically being applied to the practice, those from human factors, computer science, and psychology, for example, were essential, but not sufficient.  My belief was that the skills that had been developed over years of practice by architects, industrial designers, and graphics designers, for example, could be productively adapted to the practice of user experience design.  The resulting book tried to show how, and how such practice could fit into the other interdependent pillars of product design, namely technology and business.

The book, therefore, had to concentrate on this aspect of design practice.  Even then, it was arguably too long and too broad.  This was frustrating for me, since I knew that what I was leaving out was really important.  Furthermore, by placing so much emphasis on sketching, and the application of concepts derived from traditional design, I was worried that I might be swinging the pendulum too far in the other direction.  The key point was, the techniques derived from traditional design were meant to augment, not replace, the other disciplines being used.  I firmly believed, and still believe, in the notion of the interdependent, interwoven "renaisance team", with the triumverate of design, technology and business at its core.

Given my level of concern that by my emphasis I might be providing a distorted view of my own perspective, on pages 124 and 125 of that book, I made a kind of token hand-wave (seen in the image below), whose intent was to let the reader know that I knew that what I was writing was not the whole story.  

My intent was to say something like, "It is just as important to be able to sketch basic theoretical models of your idea on the back of an envelope, as it is to be able to sketch out some literal representation it."  The sketching book was about the latter.  That was all I had time and space to address there.  This current book is about the former.  Together, they represent what I see as two key building blocks that make up the skills of a rounded designer, or design team.

In many ways, this whole conversation is to frame what I have been attempting to do in this current manuscript, within the context of what I did in the sketching book.  This book is an attempt to capture the essence of the skills that are reflected from the back of the envelope  in the image.  What I discuss here can certainly benefit from the knowledge - such as it is - gained from the sketching book.  But by the same token, my hope is that what I discuss in this book will not only help inform the decisions made in the sketching and early design phase, it will help one start and work at a higher level by applying the theories, models, and techniques to help evaluate concepts and make more informed choices about what, where, when and how to sketch.

To me, combined the two books are kind of the ying and the yang of what I would describe as making up a reasonably rounded design professional - for this class of design, at least.  I hope that it accomplishes that, and hope that feedback from this draft manuscript contributes to it doing so.  Thanks in advance.

A Work in Progress

The following is a working draft of a book which I am (far too) slowly trying to complete.  The manuscript began as a collection of  notes, rough sketches, and reprints which I used in tutorials that I have given, mainly at past SIGCHI and SIGGRAPH Conferences.  Some sections of the manuscript remain in point form.  Others are near complete.  Some of the material has been published in the form of articles, but a lot of other material has not.

While incomplete, and full of rough spots, I am making the manuscript available on the web.  I am doing so because it has material in it that I think is of value, not available elsewhere. I undertook the initiative in order to help raise the state of the art.  It is not going to do so between now and the time of completion sitting on my hard drive where nobody can get access to it.

I hope that the quality and quantity of information helps to compensate for the unfinished nature of the work.  And, since there is no free lunch, the “price of admission”, of early access to the manuscript is my request that the reader will provide constructive comments that will help improve the final version.


  0. Preface
  1. An Introduction to Human Input to Computers
  2. An Illustrated Tour
  3. Alternative Perspectives
  4. Taxonomies of Input
  5. Case Study 1: Touch Tablets
  6. Case Study 2: Chord Keyboards
  7. Theories, Models & Basic Concepts   8. Human Performance  (Shumin Zhai)
  9. Some Representative 2D Tasks
10. A Collection of Innovative Techniques 11. Two Handed Input (Abigail Sellen, Yves Guiard & Bill Buxton)
12. 3D Input
13. Marking Interfaces
14. Gesture Driven Input (Mark Billinghurst)
15. The Future and Emerging Potential
16. References and Bibliography


When compared to displays and graphical output, studies of human-computer interaction have paid too little attention to input.  Interaction is a two-way street.  To be effective, there must be a balance of concern for both directions of communication.  Without this balance, the full potential power and benefit of the technology cannot be fulfilled.

Input is important, and deserves considerable attention.  But once this need is recognized, the intrepid investigator quickly runs into a brick wall.  Useful information relevant to the designer is near impossible to obtain.  The standard texts on computer graphics (such as Newman & Sproull, 1979, and Foley & Van Dam, 1982) give us general descriptions of devices, both physical and virtual, as well as interaction techniques.  Sherr (1988) gives a detailed and comprehensive introduction to input technologies.   There has also been a great deal of experimental work in evaluating and comparing input technologies.  Milner (1988) and Greenstein and  Arnaut (1988), for example, give a fairly comprehensive reviews of the literature.

However, the "real world" is not general.  It is made up of real problems, where the designer has to find some match among a mix of users, display technologies, tasks, context and input technologies.  Each situation is different, and requires a deep understanding of the relevant factors.  But where does one go to obtain this understanding ?

In our opinion, there is no single source, nor even any manageable small collection of sources.  While we would love to address this problem once and for all, space, time – and not least of all our own limitations - do not allow us to do so.  Thus, we will try to do the next best thing.

In what follows, it is our goal to talk - albeit briefly - about some of the main issues as we see them.  But perhaps as much time will be spent identifying the issues and how to approach them, as in giving definitive answers.  This way, our intrepid adventurer has some advance knowledge of the pit-falls and the pots of gold that lie in wait, as well the basic tools to mine them.


While I am responsible for the overall book and most of the writing, this manuscript includes significant contributions from some close colleagues.  Shumin Zhai was the primary author for Chapter 8.  Chapter 11 was coauthored with Abigail Sellen and Yves Guiard.  The primary author of chapter 10 was Mark Billinghurst.


This manuscript is a work in progress.  It has evolved over a number of years with the help of several people.  I am particularly indebted to my colleagues and students at the University of Toronto.  Ron Baecker and my Input Research Group have provided invaluable help.  Scott Mackenzie and Gordon Kurtenbach deserve particular mention.  Edgar Mathias helped uncover a number of references for me.  Bart Kip of Philips did several excellent figures.  The work of many of my graduate students is reflected in much of what follows.  The work of Ralph Hill and Brad Myers made a significant contribution to my understanding of many of the issues covered.  My colleagues at Xerox PARC have also provided a forum for discussing and working out ideas.  Stu Card and Jock Macinlay have provided many lively discussions about the material.  Bill Gaver and Sara Bly have provided valuable input on some chapters.  I would also like to thank Yutaka Hidai of Toshiba Corp for his help.

The research behind this work has been mainly funded by the Natural and Engineering Research Council of Canada.  Their ongoing support has been more than appreciated.  Additional support has also been from Rank Xerox EuroPARC and Xerox PARC, both of which have encouraged the work and enabled me to pursue it on "company time."  Likewise, Alias | Wavefront and SGI. have supported me during much of the preparation of this manuscript. Finally, much of the research from our lab at the University of Toronto has been supported by generous grants from Xerox Canada, Apple Computer and Digital Equipment Corp.  The efforts of Joy Mountford of Apple and Richard Rubenstein of DEC in making the Apple and DEC support possible is gratefully acknowledged.

Finally, it is important to mention the SIGCHI and SIGGRAPH conferences, whose conference tutorial programs provided a forum for me to develop the structure of the material.  The opportunity that that provided, and the comments of the number of tutorial attendees on the material has made a significant contribution to the form of the material.