Home   Projects   Papers   Students			Affordances and tool use Tool use is generally acknowledged as being central to intelligent behavior, rivaling language in importance to the study of cognitive phenomena. In AI and agents research, however, models of tool use, as a distinct subproblem of agent/environment interaction, have received relatively little attention. This lack of concern is reasonable from one common perspective; for an artificial agent, use of a tool may require nothing more than switching in a new set of operators for reasoning about a new effector. Nevertheless, if we are to build agents that can fully exploit a tool-oriented environment designed for humans, we must have a reasonable understanding of the nature of tool use. The development of "habile" agents, or tool-using agents, was identified by Nils Nilsson in a recent AAAI address as one of the most important open problems in current agent research. The concept of tool is difficult to define precisely, but Beck's research on non-human primate tool use is an often-cited source: "Thus tool use is the external employment of an unattached environmental object to alter more efficiently the form, position or condition of another object, another organism, or the user itself when the user holds or carries the tool during or just prior to use and is responsible for the proper and effective orientation of the tool." Our goal is be to develop a conceptual framework based on this and related characterizations, in which we can describe and differentiate specific agent activities in the interface as examples of tool use. This material is based upon work supported by the National Science Foundation under Grant No. 0083281. Related work also funded under this grant explores the topic of interface softbots, agents that act through the user interface. A summary of the goals of the project, with progress annotations, is online as well. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the National Science Foundation. Participants Robert St. Amant (PI) Ergun Bicici (MS candidate) Colin Butler (undergraduate) David Christian (MS candidate) Thomas Horton (MS candidate) Publications Robert St. Amant and Alexander B. Wood. Tool use for autonomous agents. Proceedings of the National Conference on Artificial Intelligence (AAAI). Pp. 184-189. 2005. Colin G. Butler and Robert St. Amant. HabilisDraw DT: A Bimanual Tool-Based Direct Manipulation Drawing Environment. ACM Conference on Human Factors in Computing Systems (CHI), short papers. 2004. Robert St. Amant and Thomas E. Horton. Tool-Based Direct Manipulation Environments. Under review. Robert St. Amant and Colin Butler. Demonstration: Two-handed interaction in a tool-based environment. User Interface Software and Technology (UIST) demonstration. 2003. Robert St. Amant and Thomas E. Horton. Characterizing tool use in an interactive drawing environment. Second International Symposium on Smart Graphics. 2002. Robert St. Amant and Thomas E. Horton. A tool-based interactive drawing environment. ACM Conference on Human Factors in Computing Systems (CHI) Extended Abstracts. 2002. Robert St. Amant, Henry Lieberman, Richard Potter, and Luke S. Zettlemoyer. Visual generalization in programming by example. Communications of the ACM, 43(3): 107-114. March, 2000. Robert St. Amant. User interface affordances in a planning representation. Human-Computer Interaction, 14(3): 317-354. 1999. Robert St. Amant. Planning and user interface affordances. Proceedings of Intelligent User Interfaces. 1999. Pp. 135-142. Working notes These are for internal use only; they reflect informal ideas about the directions that the project is taking at different points in time. A preliminary discussion of tools and tool use, April 25, 2002. (Also available as NCSU technical report TR-2002-06.) PBD and EUP in HabilisDraw, March 4, 2002. Basic concepts for tool use in HabilisDraw, December 6, 2001. Elements of tool use for a simulated robot arm, September 10, 2001. The use of tools, May 24, 2001. Software We have built a user interface for drawing that relies on explicit tools rather than global modes in its execution. In most graphical drawing applications, one "uses tools" for drawing lines, circles, etc., by pressing a button on an icon to specify a drawing mode, and then dragging the mouse cursor over the canvas in specific patterns. Such activity does not exploit our tool-using abilities in any significant way. The picture to the left, above, shows an alternative approach, a drawing environment that includes explicit tools such as rulers, compasses, pins, and inkwells. We believe that some applications can benefit from a richer representation of tools and tool use. When complete the HabilisDraw system will be released for general use. You can see the tools developed for Version 1, and a result of using Version 2. On the left is HabilisDraw DT, which relies on a two-handed touch interface. You can also watch movies, in AVI format, of the use of HabilisDraw 1 and 2 and of HabilisDraw DT. A small application based on the same ideas about tool use has been developed to support user control of an optimization process. A hammer, pushpins, and a marking pen are the active tools. A traveling salesman problem is represented in the box. Nodes can be added, deleted, and moved by use of the pushpin. The user taps on the side of the box to start a simulated annealing process that identifies a solution. The harder the tap, the larger the space searched. The marking pen is used to constrain those parts of a given solution that should remain fixed. Click for AVI movie. We are also working on a robot arm simulation to test our ideas about tool use in a physical environment. The simulation will represent spatial relationships between objects, dynamic change, and a controller that can use tools in the environment. The work is not yet complete, but the video link to the left should give some idea of the physics of the system when it reaches maturity.
	Last Updated:   8/14/00, 12:46:52 PM