Air Force Funds Hirshfields' Research

Student wearing EEG equipment
Student wearing EEG equipment

Researchers have analyzed how we interact with computers for many years, but most studies have been limited to examining subjective responses that are not only fraught with bias, but also lack insight into the real-time and often subtle changes that occur in a users’ mental state while working with a computer over time. Until a few years ago, non-intrusive equipment was not available for more objective analysis.


Having received a grant for $458,900 from the Air Force Office of Scientific Research to fund these newly developed tools, Stuart Hirshfield, the Stephen Harper Kirner Chair of Computer Science, and Research Associate Leanne Hirshfield ’02 have begun studying the real-time, quantitative assessment of computer users’ mental states to enhance usability testing and to create adaptive computer systems. They are creating a state-of-the-art usability laboratory that allows them to make concurrent cognitive, physiological and behavioral user measurements. They are able to monitor real-time physiological reactions with this cutting-edge equipment, including the functional near?infrared spectroscopy (fNIRS) with which they can measure users’ brain activity in realistic working conditions. With the fNIRS and other non-invasive equipment, they have begun evaluating and developing adaptive interfaces that react to user workload, moods, and emotions.

Both Hirshfields have expertise critical to this research: in usability testing, user interface design, computer programming, machine learning and cognitive psychology. Leanne Hirshfield completed her doctorate in computer science at Tufts University in 2009. Her specialty is in the area of Human-Computer Interaction (HCI), and she recently presented the first published paper on using fNIRS for usability testing.

In recent years, Stuart Hirshfield has focused his teaching and research interests on his roots in HCI. Early in his career, he worked as a research scientist for Xerox when the corporation was developing some of the first computer interfaces. He has taught courses focusing on HCI and the application of brain-computer interface technologies and has been involved in numerous research efforts that deal directly with HCI development and evaluation at the Air Force Research Laboratory in Rome, New York.
What makes their research unique is the non-invasive equipment they use to measure user state including:

• the fNIRS, a rare and cutting-edge piece of equipment which measures blood movement in the brain,
• a wireless EEG small enough to fit under a baseball cap which measures the speed with which neurons fire,
• an eye tracker, a camera that assesses work overload determining where someone is looking and measuring pupil size, and
• a device for measuring galvanic skin response.

In addition, the team employs usability software that measures more traditional remote metrics such as speed, accuracy, key strokes, mouse movement and screen captures.

One aspect of their recent research focuses on modeling and quantifying the level of trust that exists in typical interactions between human users and computer systems. This work can be applied in many ways, among them: to ensure that a given website is designed appropriately to maximize users’ trust; to gain a better understanding of the training needed to ensure that military personnel can detect system breaches quickly and accurately; to determine when an adversarial operator becomes suspicious that his or her computer has been hacked; to create systems that adapt in real time based on a user's mental state; to evaluate the effectiveness of various training and educational programs to determine what a user's level of workload is during a learning exercise; and to determine what, if any, emotional factors influence a user's comprehension. ?

The Hirshfields are engaged in several related HCI projects involving Hamilton students. This summer they worked with four students on identifying and evaluating patterns and levels of trust, suspicion and frustration with various computer interfaces while using EEG equipment as well as an fNIRS device. Sam Hincks ’11, Rachel Ward ’12, Tom Williams ’11 and Matt Russell ’11 designed and ran experiments. They are co-authors with the Hirshfields on a paper describing some of their findings, titled “This is Your Brain on Interfaces: Using Functional Near Infrared Spcctroscopy to Enhance Usability Testing.” The paper is currently under review for publication. This fall Leanne Leanne is continuing her work measuring user states. One of her goals is to quantify changing levels of trust toward computer systems, focusing on how to interpret brain data, developing and experimenting with various algorithms to determine which are optimal for interpreting the cognitive and emotional levels. Additionally, she is continuing research to measure user states that are related to user preference, frustration, enjoyment, and surprise, as measurement of these user states would be very useful during usability tests.

Stuart Hirshfield is teaching a course this semester to both computer science and neuroscience majors focusing on the application of these techniques to HCI. Using objective data, students are evaluating the Blackboard course management system versus alternate course delivery systems. Hirshfield hopes to engage in similar usability evaluation projects for government and commercial entities in the future. He has also applied for National Science Foundation funding for a human computer interface project through the Utica schools with middle school students in the summer as a way of engaging them in computer science.

More information about the lab and the Hirshfields’ research can be found at the website for the usability lab: http://usabilitylab.hamilton.edu.

Back to Top