I. SegMan

From Robert St. Amant and Mark O. Riedl (2001), A perception/action substrate for cognitive modeling in HCI, International Journal of Human-Computer Studies, 55:15-39.  (Draft at http://www.csc.ncsu.edu/faculty/stamant/papers/):     

An interface softbot, unlike the current generation of interface agents, controls an interactive system through the graphical user interface, as human users do, without relying on an application programming interface (API) or access to source code.  To support this functionality in the Microsoft Windows environment we have developed a programmable substrate that we call VisMap, for "Visual manipulation."  In VisMap, sensors takes pixel-level input from the display, runs the data through image processing algorithms, and builds a structured representation of visible interface objects.  Effectors generates mouse and keyboard gestures to manipulate these objects.  The sensors and effectors act as the eyes and hands of an artificial user, to be controlled externally by a computational cognitive model.

The version you've downloaded is a descendant of the one described in this paper.  We've named the new system SegMan, for "segmentation and manipulation."

II. Installing SegMan

First you will need a Common Lisp environment that can load and execute foreign (C) .dll files.  We use Allegro Common Lisp from Franz and have designed SegMan to work with its foreign function interface. Porting the system to Lispworks from Xanalys would be elementary, based on our experience with VisMap, but this has not yet been done for the new system.

Download the file segman.zip, and if you'd like more than the basic functionality, segman-aux.zip.

To install SegMan, unzip segman.zip.  This will create a new folder (directory) on your system called "segman", which contains the files needed to run the substrate.  If you'd like to see SegMan working with a planning system or a cognitive model, also unzip segman-aux.zip, which will create several more folders, including RAP-System, UCPOP, DHP, and a relatively old version of ACT-R.  All of these folders need to be kept in the same folder for the system to load properly.  At NCSU, we follow a convention of storing all such Lisp system folders in c:\\research\\systems\\*.

For SegMan to work with Windows, a few system parameters need to be set up appropriately.  First try running segmanfontinit.exe, found in the installation/font adjustments subdirectory.  This will set the system font to MS Sans Serif, 8 point, which the optical character recognition algorithms in SegMan require.  If this does not work, for whatever reason, then you can do this part manually, from the start menu:  Start -> Settings -> Display control panel.  In the Appearance tab, click on the "Message Text" and set the font to "MS Sans Serif", 8 point.  Click on the "Selected" menu header and set it to the same.

A few more things need to be taken care of.

Make sure that the task bar (the bar at the bottom of the screen with the Start button) is at the bottom of the screen rather than the top: Start -> Settings -> Taskbar & Start Menu.  If the task bar is at the top of the screen, the system may not be able to detect the appearance of the Start menu (because of border overlaps with running application buttons to the right of the Start button.)  It's most convenient, though not always necessary, if the start button is always visible.  

Make sure that any applications that SegMan is intended for use the MS Sans Serif, 8 point, font; otherwise the system will not be able to read the menus.  

Some screen background colors (e.g. black) interfere with the segmentation of window borders, and so prevent windows from being identified correctly.  Use a standard color scheme in which borders of objects are visible.  

If you have trouble using SegMan within a Microsoft application, it may be because the application is not properly "aware" of the font it should use.  Instead of MS Sans Serif, try VisMap 8, a font you can find in vismap.fon, in the installation/font adjustments subdirectory.

We are currently working on eliminating such restrictions, but this will take time.  We've built the system relying on visual properties of the interface that are pretty much in an out-of-the-box state.

III. Using SegMan

To load and run SegMan, start up Allegro Common Lisp (with the IDE or without) and load load-system.lisp, which is in the segman folder. All the files that are required to run SegMan should load automatically.  The first time you do this, the system may need to create new directories to contain Lisp binaries.  The system tries to do this automatically, but it doesn't always work; if it does not, you'll have to create the directories by hand, looking at the error message to give you the name of each directory.

Mark Riedl has written a relatively extensive description of how the system functions.  His documentation can be found in segman-introduction.htm, in the installation subdirectory.

We have written a few demo programs that use SegMan.  (For these demos to work, make sure that the Notepad application is visible when the Start menu is first raised.)  Type the following into the Allegro console:

    (user::load-test-domain)
    (user::notepad-demo)

The Notepad demo opens the Start menu and searches through the submenus for the Notepad application.  When Notepad opens, the program uses the application's menu to open a file called "notepad-demo.txt" in the SegMan directory.  The file has the word "three" repeated twice; the program highlights the first occurence it "sees" and uses the application's menus to cut the duplicate out.  The demo program ends when the word "three" is finally cut.

If you'd loaded the segman-aux material, you can try typing these forms into the Allegro console:

    (user::load-actr-domain)
    (user::actr-demo) 

This loads a version of Mike Byrne's ACT-R/PM, based on ACT-R 4.0 and ACT-R/PM 1.0b5, a release from late 1999.  It has been modified, or rather extended, to substitute SegMan functionality for the MCL-specific hooks.  See mcl-simulated-interface.lisp in the ACTR Code directory to see alternative functionality for mcl-interface.lisp. The demo is a simplified version of one of Mike Byrne's examples.  It starts up Notepad, types in a line of text "The boy hit the dog" and then runs a simple ACT-R model to find and delete the nouns in that sentence.  Sometimes it does not complete the process, but we're not entirely sure why.  It would be very nice to find more sophisticated ACT-R models to work with, but we're not cognitive modelers (yet).