Publications

Smart glasses with AI assistants are increasingly used in daily life. However, current systems lack awareness of the user's internal cognitive state, leaving them unable to proactively anticipate users' needs without access to cognitive load. Existing methods for assessing cognitive load either rely on impractical sensors for lightweight eyewear or utilize eye gaze-based models that suffer from poor interpretability, and require task-specific fine-tuning, often failing to generalize across individuals.

We propose GazeMind, a gaze-guided LLM agent framework for cognitive load assessment on smart glasses. It encodes eye-tracking data into structured representations for LLM-based reasoning and provides interpretable cognitive load predictions. Importantly, GazeMind generalizes across scenarios without LLM fine-tuning through a novel task-guidance reasoning approach and achieves personalized adaptation by incorporating user-specific characteristics and historical references. To support evaluation, we introduce CogLoad-Bench, the largest gaze-based cognitive load dataset with 152 participants, 40+ hours of multimodal data, and 10K+ real-time annotations across controlled and real-world tasks. Experiments show that GazeMind achieves state-of-the-art performance, outperforming baselines by over 20% across all metrics.

@article{wang2026gazemind,
author = {Wang, Bin and Liu, Yue and Newman, Benjamin and Fernandes, Ajoy S. and Wang, Zhiyuan and Cavin, Robert and Cox, Michele A. and Rajanna, Vijay and Bolte, Takumi and Hunfalvay, Melissa and Bagci, Ulas and Proulx, Michael J.},
title = {GazeMind: A Gaze-Guided LLM Agent for Personalized Cognitive Load Assessment},
journal = {arXiv preprint arXiv:2605.05790},
year = {2026},
month = {May},
eprint = {2605.05790},
archivePrefix = {arXiv},
primaryClass = {cs.HC}
}

Text entry is extremely difficult or sometimes impossible in the scenarios of situationally-induced or physical impairments and disabilities. As a remedy, many rely on gaze typing which commonly uses dwell time as the selection method. However, dwell-based gaze typing could be limited by usability issues, reduced typing speed, high error rate, steep learning curve, and visual fatigue with prolonged usage.

We present a dwell-free, multimodal approach to gaze typing where the gaze input is supplemented with a foot input modality. In this multi-modal setup, the user points her gaze at the desired character, and selects it with the foot input. We further investigated two approaches to foot-based selection, a foot gesture-based selection and a foot press-based selection, which are compared against the dwell-based selection.

We evaluated our system through three experiments involving 51 participants, where each experiment used one of the three target selection methods: dwell-based, foot gesture-based, and foot press-based selection. We found that foot-based selection at least matches, and likely improves, the gaze typing performance compared to dwell-based selection. Among the four foot gestures (toe tapping, heel tapping, right flick and left flick) we used in the study, toe tapping is the most preferred gesture for gaze typing. Furthermore, when using foot-based activation users quickly develop a rhythm in focusing at a character with gaze and selecting it with the foot. This familiarity reduces errors significantly. Overall, based on both typing performance and qualitative feedback the results suggest that gaze and foot-based tying is convenient, easy to learn, and addresses the usability issues associated with dwell-based typing. We believe, our findings would encourage further research in leveraging a supplemental foot input in gaze typing, or in general, would assist in the development of rich foot-based interactions.

@article{RAJANNA2022102787,
title = {PressTapFlick: Exploring a gaze and foot-based multimodal approach to gaze typing},
journal = {International Journal of Human-Computer Studies},
volume = {161},
pages = {102787},
year = {2022},
issn = {1071-5819},
doi = {https://doi.org/10.1016/j.ijhcs.2022.102787},
url = {https://www.sciencedirect.com/science/article/pii/S1071581922000167},
author = {Vijay Rajanna and Murat Russel and Jeffrey Zhao and Tracy Hammond},
keywords = {Gaze typing, Multimodal interaction, Foot-based interaction, Virtual keyboard, Optikey, 3D printing, Microcontroller} }

Gaze input has been a promising substitute for mouse input for point and select interactions. Individuals with severe motor and speech disabilities primarily rely on gaze input for communication. Gaze input also serves as a hands-free input modality in the scenarios of situationally-induced impairments and disabilities (SIIDs). Hence, the performance of gaze input has often been compared to mouse input through standardized performance evaluation procedure like the Fitts' Law. With the proliferation of touch-enabled devices such as smartphones, tablet PCs, or any computing device with a touch surface, it is also important to compare the performance of gaze input to touch input.

In this study, we conducted ISO 9241-9 Fitts' Law evaluation to compare the performance of multimodal gaze and foot-based input to touch input in a standard desktop environment, while using mouse input as the baseline. From a study involving 12 participants, we found that the gaze input has the lowest throughput (2.55 bits/s), and the highest movement time (1.04 s) of the three inputs. In addition, though touch input involves maximum physical movements, it achieved the highest throughput (6.67 bits/s), the least movement time (0.5 s), and was the most preferred input. While there are similarities in how quickly pointing can be moved from source to target location when using both gaze and touch inputs, target selection consumes maximum time with gaze input. Hence, with a throughput that is over 160% higher than gaze, touch proves to be a superior input modality.

@ARTICLE{fitts_gaze_touch_mouse_rajanna22,
doi = {10.48550/ARXIV.2208.01248},
url = {https://arxiv.org/abs/2208.01248},
author = {Rajanna, Vijay and Hammond, Tracy},
title = {Can Gaze Beat Touch? A Fitts' Law Evaluation of Gaze, Touch, and Mouse Inputs},
publisher = {arXiv},
year = {2022} }

Despite the utility of gaze gestures as an input method, there is a lack of guidelines available regarding how to design gaze gestures, what algorithms to use for gaze gesture recognition, and how these algorithms compare in terms of performance. To facilitate the development of applications that leverage gaze gestures, we have evaluated the performance of a combination of template-based and data-driven algorithms on two custom gesture sets that can map to user actions. Template-based algorithms had consistently high accuracies but the slowest runtimes, making them best for small gesture sets or accuracy-critical applications. Data-driven algorithms run much faster and scale better to larger gesture sets, but require more training data to achieve the accuracy of the template-based methods.

@ARTICLE{9663039,
author={Li, Jiayao and Ray, Samantha and Rajanna, Vijay and Hammond, Tracy},
journal={IEEE Access},
title = {Evaluating the Performance of Machine Learning Algorithms in Gaze Gesture Recognition Systems},
year = {2022},
volume={10},
pages={1020-1035},
doi={10.1109/ACCESS.2021.3136153} }

This paper presents a Fitts' law experiment and a clinical case study performed with a head-mounted display (HMD). The experiment compared gaze, foot, and head pointing. With the equipment setup we used, gaze was slower than the other pointing methods, especially in the lower visual field. Throughputs for gaze and foot pointing were lower than mouse and head pointing and their effective target widths were also higher. A follow-up case study included seven participants with movement disorders. Only two of the participants were able to calibrate for gaze tracking but all seven could use head pointing, although with throughput less than one-third of the non-clinical participants.

@inproceedings{10.1145/3317956.3318150,
author = {Minakata, Katsumi and Hansen, John Paulin and MacKenzie, I. Scott and B\ae{}kgaard, Per and Rajanna, Vijay},
title = {Pointing by Gaze, Head, and Foot in a Head-Mounted Display},
year = {2019},
publisher = {ACM},
doi = {10.1145/3317956.3318150},
booktitle = {Proceedings of the 11th ACM Symposium on Eye Tracking Research & Applications},
series = {ETRA ’19} }

We investigate new media to improve how teams of students create and organize artifacts as they perform design. Some design artifacts are readymade—e.g., prior work, reference images, code framework repositories—while others are self-made—e.g., storyboards, mock ups, prototypes, and user study reports. We studied how computer science students use the medium of free-form web curation to collect, assemble, and report on their team-based design projects. From our mixed qualitative methods analysis, we found that the use of space and scale was central to their engagement in creative processes of communication and contextualization.

@inproceedings{10.1145/3325480.3325483,
author = {Lupfer, Nic and Kerne, Andruid and Linder, Rhema and Fowler, Hannah and Rajanna, Vijay and Carrasco, Matthew and Valdez, Alyssa},
title = {Multiscale Design Curation: Supporting Computer Science Students’ Iterative and Reflective Creative Processes},
year = {2019},
publisher = {ACM},
doi = {10.1145/3325480.3325483},
booktitle = {Proceedings of the 2019 Conference on Creativity and Cognition},
series = {C&C ’19} }

Every day we encounter a variety of scenarios that lead to situationally induced impairments and disabilities, i.e., our hands are assumed to be engaged in a task, and hence unavailable for interacting with a computing device. For example, a surgeon performing an operation, a worker in a factory with greasy hands or wearing thick gloves, a person driving a car, and so on all represent scenarios of situational impairments and disabilities. In such cases, performing point-and-click interactions, text entry, or authentication on a computer using conventional input methods like the mouse, keyboard, and touch is either inefficient or not possible. Unfortunately, individuals with physical impairments and disabilities are forced to deal with these limitations every single day. Therefore, to address situational and physical impairments and disabilities it is crucial to develop hands-free, accessible interactions.

In this research, we address these limitations by developing a gaze-assisted, multi-modal, hands-free, accessible interaction paradigm. Specifically, we focus on three primary interactions: 1) point-and-click, 2) text entry, and 3) authentication.

Rajanna, Vijay Dandur (2018). Addressing Situational and Physical Impairments and Disabilities with a Gaze-Assisted, Multi-Modal, Accessible Interaction Paradigm. Doctoral dissertation, Texas A&M University.

Gaze tracking in virtual reality (VR) allows for hands-free text entry, but it has not yet been explored. We investigate how the keyboard design, selection method, and motion in the field of view may impact typing performance and user experience. We present two studies of people (N=32) typing with gaze+dwell and gaze+click inputs in VR.

@inproceedings{ETRA18:VRGazeTyping,
author = {Rajanna, Vijay and Hansen, John Paulin},
title = {Gaze Typing in Virtual Reality: Impact of Keyboard Design, Selection Method, and Motion},
booktitle = {Proceedings of the Tenth Biennial ACM Symposium on Eye Tracking Research and Applications},
series = {ETRA '18},
year = {2018},
doi = {10.1145/3204493.3204541} }

Gaze-assisted interaction has commonly been used in a standard desktop setting. When interacting with large displays, as new scenarios like situationally-induced impairments emerge, it is more convenient to use the gaze-based multimodal input than other inputs. We compared gaze+foot multimodal input to touch and mouse inputs on a large display in a Fitts' Law experiment that conforms to ISO 9241-9.

@inproceedings{COGAIN18:FittsLargeDisplay,
author = {Rajanna, Vijay and Hammond, Tracy},
title = {A Fitts' Law Evaluation of Gaze Input on Large Displays Compared to Touch and Mouse Inputs},
booktitle = {COGAIN '18: Workshop on Communication by Gaze Interaction, June 14--17, 2018, Warsaw, Poland},
series = {COGAIN '18},
year = {2018},
doi = {10.1145/3206343.3206348} }

Gaze and head tracking, or pointing, in head-mounted displays enables new input modalities for point-select tasks. We conducted a Fitts' law experiment with 41 subjects comparing head pointing and gaze pointing using a 300 ms dwell (n = 22) or click (n = 19) activation, with mouse input providing a baseline for both conditions.

@inproceedings{COGAIN18:FittsVR,
author = {Hansen, John Paulin and Rajanna, Vijay and MacKenzie, I. Scott and B\ae kgaard, Per},
title = {A Fitts' Law Study of Click and Dwell Interaction by Gaze, Head and Mouse with a Head-Mounted Display},
booktitle = {COGAIN '18: Workshop on Communication by Gaze Interaction},
year = {2018},
doi = {10.1145/3206343.3206344} }

Gaze gesture-based interactions on a computer are promising, but the existing systems are limited by the number of supported gestures, recognition accuracy, need to remember the stroke order, lack of extensibility, and so on. We present a gaze gesture-based interaction framework where a user can design gestures and associate them to appropriate commands.

@inproceedings{ETRA2018:Gazegestures,
author = {Rajanna, Vijay and Hammond, Tracy},
title = {A Gaze Gesture-based Paradigm for Situational Impairments, Accessibility, and Rich Interactions},
booktitle = {Proceedings of the Tenth Biennial ACM Symposium on Eye Tracking Research and Applications},
series = {ETRA '18},
year = {2018},
doi = {10.1145/3204493.3208344} }

Shoulder surfing enables an attacker to gain the authentication details of a victim through observations and is becoming a threat to visual privacy. We present DyGazePass: Dynamic Gaze Passwords, an authentication strategy that uses dynamic gaze gestures. We also present two authentication interfaces, a dynamic and a static-dynamic interface, that leverage this strategy to counter shoulder surfing attacks.

@INPROCEEDINGS{8311458,
author={V. Rajanna and A. H. Malla and R. A. Bhagat and T. Hammond},
booktitle={2018 IEEE 4th International Conference on Identity, Security, and Behavior Analysis (ISBA)},
title={DyGazePass: A gaze gesture-based dynamic authentication system to counter shoulder surfing and video analysis attacks},
year={2018},
doi={10.1109/ISBA.2018.8311458} }

Artists value the ability to determine what parts of their composition is most appreciated by viewers. This information normally comes straight from viewers in the form of oral and written feedback. To address this issue, we developed a web-based system where artists can upload their artwork to be viewed by the viewers on their computer while a web camera tracks their eye movements.

Failing to brush one's teeth regularly can have surprisingly serious health consequences, from periodontal disease to coronary heart disease to pancreatic cancer. We developed a non-invasive wearable system based on a wrist-mounted accelerometer to accurately identify when a person brushed their teeth, achieving an accuracy of 94%.

@inproceedings{Cherian2017pervasive,
author = {Cherian, Josh and Rajanna, Vijay and Goldberg, Daniel and Hammond, Tracy},
title = {Did you Remember To Brush? : A Noninvasive Wearable Approach to Recognizing Brushing Teeth for Elderly Care},
booktitle = {11th EAI International Conference on Pervasive Computing Technologies for Healthcare},
year = {2017} }

Shoulder-surfing is the act of spying on an authorized user of a computer system with the malicious intent of gaining unauthorized access. We present an intelligent gaze gesture-based system that authenticates users from their unique gaze patterns onto moving geometric shapes. In a study with 15 users, authentication accuracy was found to be 99%.

@inproceedings{Rajanna:2017:GGU:3027063.3053070,
author = {Rajanna, Vijay and Polsley, Seth and Taele, Paul and Hammond, Tracy},
title = {A Gaze Gesture-Based User Authentication System to Counter Shoulder-Surfing Attacks},
booktitle = {Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems},
series = {CHI EA '17},
year = {2017},
doi = {10.1145/3027063.3053070} }

Recent advancements in eye tracking technology are driving the adoption of gaze-assisted interaction as a rich and accessible human-computer interaction paradigm. This dissertation research proposes a gaze-assisted, multimodal interaction paradigm, and related frameworks and their applications that effectively enable gaze-assisted interactions while addressing many of the current limitations.

@misc{https://doi.org/10.48550/arxiv.1803.04713,
doi = {10.48550/ARXIV.1803.04713},
url = {https://arxiv.org/abs/1803.04713},
author = {Rajanna, Vijay and Hammond, Tracy},
title = {A Gaze-Assisted Multimodal Approach to Rich and Accessible Human-Computer Interaction},
year = {2018} }

Gaze Typing allows individuals with motor impairments to enter text on a computer using a virtual keyboard and their gaze. We present a gaze-assisted, wearable-supplemented, foot interaction framework for dwell-free gaze typing using a custom-built virtual keyboard, eye tracker, and foot wearable.

@inproceedings{Rajanna:2016:GTT:2982142.2982145,
author = {Rajanna, Vijay},
title = {Gaze Typing Through Foot-Operated Wearable Device},
booktitle = {Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility},
series = {ASSETS '16},
year = {2016},
doi = {10.1145/2982142.2982145} }

Shoulder-surfing is an act of spying when an authorized user is logging into a system. We present a gaze-assisted user authentication system as a potential solution to counter shoulder-surfing attacks using moving shapes and scan-path pattern recognition.

In this research, we present an eye tracking framework that, based on eye movement data, can assess students' perceived activities and overall engagement in a sketch based Intelligent tutoring system ("Mechanix") with 97.12% accuracy.

@inproceedings{Kaul:2016:EUP:2931002.2948727,
author = {Kaul, Purnendu and Rajanna, Vijay and Hammond, Tracy},
title = {Exploring Users' Perceived Activities in a Sketch-based Intelligent Tutoring System Through Eye Movement Data},
booktitle = {Proceedings of the ACM Symposium on Applied Perception},
series = {SAP '16},
year = {2016},
doi = {10.1145/2931002.2948727} }

We present CANE: "Computer Assisted Navigation Engine," a low cost, wearable, and haptic-assisted navigation smart belt system for the visually impaired providing vibration feedback around the waist.

@inproceedings{polsley2016cane,
title={CANE: A Wearable Computer-Assisted Navigation Engine for the Visually Impaired},
author={Polsley, Seth and Rajanna, Vijay and Powell, Larry and Tapie, Kodi and Hammond, Tracy},
booktitle={Workshop on Smart Connected and Wearable Things 2016} }

GAWSCHI is a Gaze-Augmented, Wearable-Supplemented Computer-Human Interaction framework enabling accurate and quick gaze-driven interactions with a foot-operated quasi-mouse.

@inproceedings{Rajanna:2016:GGW:2857491.2857499,
author = {Rajanna, Vijay and Hammond, Tracy},
title = {GAWSCHI: Gaze-augmented, Wearable-supplemented Computer-human Interaction},
booktitle = {Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research and Applications},
series = {ETRA '16},
year = {2016},
doi = {10.1145/2857491.2857499} }

Transforming gaze input into a rich and assistive interaction modality through combining gaze and foot interaction modalities for precise point-and-click operations.

@inproceedings{Rajanna:2016:GFI:2876456.2876462,
author = {Rajanna, Vijay Dandur},
title = {Gaze and Foot Input: Toward a Rich and Assistive Interaction Modality},
booktitle = {Companion Publication of the 21st International Conference on Intelligent User Interfaces},
series = {IUI '16 Companion},
year = {2016},
doi = {10.1145/2876456.2876462} }

KinoHaptics is an automated, wearable, haptic-assisted physio-therapeutic system that provides vibro-haptic feedback to support post-surgery rehabilitation, prevent accidental injuries, and encourage habit building.

@Article{Rajanna2015,
author="Rajanna, Vijay and Vo, Patrick and Barth, Jerry and Mjelde, Matthew and Grey, Trevor and Hammond, Tracy",
title="KinoHaptics: An Automated, Wearable, Haptic Assisted, Physio-therapeutic System for Post-surgery Rehabilitation and Self-care",
journal="Journal of Medical Systems",
year="2015",
doi="10.1007/s10916-015-0391-3" }

Let Me Relax is a fully automated sedentary-state recognition framework using smartwatch and smartphone sensors to encourage mental wellness through interventions in the form of relaxation techniques.

@inproceedings{Rajanna:2015:LMR:2897442.2897461,
author = {Rajanna, Vijay and Alamudun, Folami and Goldberg, Daniel and Hammond, Tracy},
title = {Let Me Relax: Toward Automated Sedentary State Recognition and Ubiquitous Mental Wellness Solutions},
booktitle = {Proceedings of the 5th EAI International Conference on Wireless Mobile Communication and Healthcare},
year = {2015},
doi = {10.4108/eai.14-10-2015.2261900} }

Step Up Life is a context-aware mobile health assistant application providing physical activity reminders considering user location, personal preferences, calendar events, time of day, and weather.

@inproceedings{Rajanna:2014:SUL:2676629.2676636,
author = {Rajanna, Vijay and Lara-Garduno, Raniero and Behera, Dev Jyoti and Madanagopal, Karthic and Goldberg, Daniel and Hammond, Tracy},
title = {Step Up Life: A Context Aware Health Assistant},
booktitle = {Proceedings of the Third ACM SIGSPATIAL International Workshop on the Use of GIS in Public Health},
year = {2014},
doi = {10.1145/2676629.2676636} }

A generic accelerometer-based gesture recognition framework providing natural and intuitive remote interactions with desktop applications over Bluetooth.

@article{rajanna2013framework,
title={Framework for accelerometer based gesture recognition and seamless integration with desktop applications},
author={Rajanna, Vijay D},
journal={International Journal of Scientific and Research Publications},
year={2013} }