Academic Research

Foot gesture recognizer. From left to right (1) Foot Gesture Recognition Device - Master unit: the entire circuitry of the master unit is housed inside a3D-printed container that is attached to the user’s footware. The user is executing a toe tap gesture (2) Foot Gesture Recognition Device: the master and receiver units. The master unit is attached to user’s footware, and the receiver unit is connected to the computer through USB port, and 3) Gesture Recognition: the list of foot gestures that are recognized by the device. The angular velocities indicate the speed at which the user has to tap or flick the foot to trigger a corresponding gesture.

Text entry is extremely difficult or sometimes impossible in the scenarios of situationally-induced or physical impairments and disabilities. As a remedy, many rely on gaze typing which commonly uses dwell time as the selection method. However, dwell-based gaze typing could be limited by usability issues, reduced typing speed, high error rate, steep learning curve, and visual fatigue with prolonged usage. We present a dwell-free, multimodal approach to gaze typing where the gaze input is supplemented with a foot input modality. In this multi-modal setup, the user points her gaze at the desired character, and selects it with the foot input. We further investigated two approaches to foot-based selection, a foot gesture-based selection and a foot press-based selection, which are compared against the dwell-based selection.

Enhanced Virtual Keyboard:

Foot Press Sensing Device: 1) Foot Press Sensing device attached to the user’s footware. The pressure sensor is placed inside the footware. Foot Press Sensing Device: the entire circuitry is housed inside a 3D-printed container, and the force sensitive resister that senses foot press actions extends from the main circuit and is placed inside the footware. 2) An outline of how the foot press sensing device is attached to the user’s footware, and the placement of the pressure sensor inside the footware.

Fitts' Law Experiment: a participant performing a multi-directional point-and-select Fitts' Law task. From left to right (1) Touch Input, 2) Mouse Input, and 3) Gaze Input.

Gaze input has been a promising substitute for mouse input for point and select interactions. Individuals with severe motor and speech disabilities primarily rely on gaze input for communication. Gaze input also serves as a hands-free input modality in the scenarios of situationally-induced impairments and disabilities (SIIDs). Hence, the performance of gaze input has often been compared to mouse input through standardized performance evaluation procedure like the Fitts' Law. With the proliferation of touch-enabled devices such as smartphones, tablet PCs, or any computing device with a touch surface, it is also important to compare the performance of gaze input to touch input.

In this study, we conducted ISO 9241-9 Fitts' Law evaluation to compare the performance of multimodal gaze and foot-based input to touch input in a standard desktop environment, while using mouse input as the baseline. From a study involving 12 participants, we found that the gaze input has the lowest throughput (2.55 bits/s), and the highest movement time (1.04 s) of the three inputs. In addition, though touch input involves maximum physical movements, it achieved the highest throughput (6.67 bits/s), the least movement time (0.5 s), and was the most preferred input. While there are similarities in how quickly pointing can be moved from source to target location when using both gaze and touch inputs, target selection consumes maximum time with gaze input. Hence, with a throughput that is over 160% higher than gaze, touch proves to be a superior input modality.

Visualization of the paths on the screen along which the cursor traversed during touch, mouse, and gaze inputs respectively.

Fitts' Law Experiment: a participant, in an upright stance, performing a multi-directional point-and-select Fitts' Law task (1) shown on a large display. Also, an eye tracker is mounted on a tripod (2).

Left Image - Microsoft Surface Hub (84-inch). Right Image - The foot controller used in the gaze+foot selection method. 1 - a force sensitive resistor, microcontroller, and bluetooth module in a 3D printed case, 2 - foot interaction.

A user minimizing the browser with a gaze gesture.

A user draws a gesture with their eye movements, and assigns a dedication action (e.g., minimize).

The following three images show the password selection interface, dynamic interface, and static-dynamic interfces.

If a user has selected Square-Star-Pie as a password, then the user is authenticated by following each shapes' paths in their respective frames, as shown in the sequence of Figures below. The user first follows the square shape, then star, and finally pie. The user does not receive any feedback, since the gaze point and scan-path are hidden.

Users with physical impairments are limited by their ability to work on computers using the conventional mouse- and keyboard-based interactions. Existing accessible technologies still have usability issues, need a lot of training, and are imprecise. We present a gaze gesture-based interaction paradigm for users with physical impairments to work on a computer by just using their eye movements. We use an eye tracker that tracks the user's eye movements. To perform an action like minimizing and maximizing an application; opening a new tab, scrolling down, refresh on a browser, etc., the user moves their eyes to make a predefined gesture. The system recognizes the gesture performed and executes the corresponding action. Users with speech impairment can also use this system to speak quick phrases by performing gestures. This is crucial when a person with speech impairment is interacting with another person who does not know sign language.

Gaze Typing, a gaze-assisted text entry method, allows individuals with motor (arm, spine) impairments to enter text on a computer using a virtual keyboard and their gaze. Though gaze typing is widely accepted, this method is limited by its lower typing speed, higher error rate, and the resulting visual fatigue, since dwell-based key selection is used. In this research, we present a gaze-assisted, wearable-supplemented, foot interaction framework for dwell-free gaze typing. The framework consists of a custom-built virtual keyboard, an eye tracker, and a wearable device attached to the user's foot. To enter a character, the user looks at the character and selects it by pressing the pressure pad, attached to the wearable device, with the foot. Results from a preliminary user study involving two participants with motor impairments show that the participants achieved a mean gaze typing speed of 6.23 Words Per Minute (WPM). In addition, the mean value of Key Strokes Per Character (KPSC) was 1.07 (ideal 1.0), and the mean value of Rate of Backspace Activation (RBA) was 0.07 (ideal 0.0). Furthermore, we present our findings from multiple usability studies and design iterations, through which we created appropriate affordances and experience design of our gaze typing system.

A highly secured, foolproof user authentication is still a primary focus of research in the field of User Privacy and Security. Shoulder-surfing is an act of spying when an authorized user is logging into a system; it is promoted by a malicious intent of gaining unauthorized access. We present a gaze-assisted user authentication system as a potential solution counter shoulder-surfing attacks. The system comprises of an eye tracker and an authentication interface with 12 pre-defined shapes (e.g., triangle, circle, etc.) that move on the screen. A user chooses a set of three shapes as a password. To authenticate, the user follows paths of the three shapes as they move, one on each frame, over three consecutive frames. The system uses a template matching algorithms to compare the scan-path of the user's gaze with the path traversed by the shape. The system evaluation involving seven users showed that the template matching algorithm achieves an accuracy of 95%. Our study also shows that Gaze-driven authentication is a foolproof system against shoulder-surfing attacks; the unique pattern of eye movements for each individual makes the system hard to break into.

Recent developments in eye tracking technology are paving the way for gaze-driven interaction as the primary interaction modality. Despite successful efforts, existing solutions to the ``Midas Touch" problem have two inherent issues: 1) lower accuracy, and 2) visual fatigue that are yet to be addressed. In this work we present GAWSCHI: a Gaze-Augmented, Wearable-Supplemented Computer-Human Interaction framework that enables accurate and quick gaze-driven interactions, while being completely immersive and hands-free. GAWSCHI uses an eye tracker and a wearable device (quasi-mouse) that is operated with the user's foot, specifically the big toe. The system was evaluated with a comparative user study involving 30 participants, with each participant performing eleven predefined interaction tasks (on MS Windows 10) using both mouse and gaze-driven interactions. We found that gaze-driven interaction using GAWSCHI is as good (time and precision) as mouse-based interaction as long as the dimensions of the interface element are above a threshold (0.60" x 0.51"). In addition, an analysis of NASA Task Load Index post-study survey showed that the participants experienced low mental, physical, and temporal demand; also achieved a high performance. We foresee GAWSCHI as the primary interaction modality for the physically challenged and a means of enriched interaction modality for the able-bodied demographics.

Intelligent tutoring systems (ITS) empower instructors to make teaching more engaging by providing a platform to tutor, deliver learning material, and to assess students' progress. Despite the advantages, existing ITS do not automatically assess how students engage in problem solving? How do they perceive various activities? and How much time they spend on each activity leading to the solution? In this research, we present an eye tracking framework that, based on eye movement data, can assess students' perceived activities and overall engagement in a sketch based Intelligent tutoring system, "Mechanix." Based on an evaluation involving 21 participants, we present the key eye movement features, and demonstrate the potential of leveraging eye movement data to recognize students' perceived activities, "reading, gazing at an image, and problem solving," with an accuracy of 97.12%.
First Author: Purnendu Kaul

Problem Statement: 

A carefully planned, structured, and supervised physiotherapy program, following a surgery, is crucial for the successful diagnosis of physical injuries. Nearly 50% of the surgeries fail due to unsupervised and erroneous physiotherapy. The demand for a physiotherapist for an extended period is expensive to afford, and sometimes inaccessible. With the advancements in wearable sensors and motion tracking, researchers have tried to build affordable, automated, physio-therapeutic systems, which direct a physiotherapy session by providing audio-visual feedback on patient’s performance. There are many aspects of automated physiotherapy program which are yet to be addressed by the existing systems: wide variety of patients’ physiological conditions to be diagnosed, demographics of the patients (blind, deaf, etc.,), and pursuing them to adopt the system for an extended period for self-care

In our research, we have tried to address these aspects by building a health behavior change support system called KinoHaptics, for post-surgery rehabilitation. KinoHaptics is an automated, persuasive, haptic assisted, physio-therapeutic system that can be used by a wide variety of demographics and for various patients’ physiological conditions. The system provides rich and accurate vibro-haptic feedback that can be felt by any user irrespective of the physiological limitations. KinoHaptics is built to ensure that no injuries are induced during the rehabilitation period. The persuasive nature of the system allows for personal goal-setting, progress tracking, and most importantly lifestyle compatibility.

As people in industrialized countries enjoy modern conveniences that lead to greater sedentary lifestyles and decreased involvement in physical activities, they also increase their risk of acquiring hypokinetic diseases such as obesity and heart disease that negatively impact their long-term health. While emerging wearable computing technologies are encouraging researchers and developers to create fitness-based mobile user interfaces for combating the effects of sedentary lifestyles, existing solutions instead primarily cater to fitness-minded users who wish to take advantage of technology to enhance their self-motivated physical exercises.

Advances in ubiquitous computing technology improve workplace productivity, reduce physical exertion, but ultimately result in a sedentary work style. Sedentary behavior is associated with an increased risk of stress, obesity, and other health complications. Let Me Relax is a fully automated sedentary-state recognition framework using a smartwatch and smartphone, which encourages mental wellness through interventions in the form of simple relaxation techniques. The system was evaluated through a comparative user study of 22 participants split into a test and a control group. An analysis of NASA Task Load Index pre- and post- study survey revealed that test subjects who followed relaxation methods, showed a trend of both increased activity as well as reduced mental stress. Reduced mental stress was found even in those test subjects that had increased inactivity. These results suggest that repeated interventions, driven by an intelligent activity recognition system, is an effective strategy for promoting healthy habits, which reduce stress, anxiety, and other health risks associated with sedentary workplaces.

This project is a custom implementation of "Linux Shell" that enhances the basic functionalities provided by default Linux shells like Bourne, Korn, C, and Bash. The custom Linux shell thus developed is able to run executable statements with command line arguments.

How does it work ?

• On logging into the terminal, the custom shell displays the Linux prompt  indicating that it is  ready to receive a command from the user.
• The user issues a command, for example: ls <directory-name>
• The custom shell then,
• Reads the command, 
• Searches for and locates the file with that name in the directories containing utilities, 
• Loads the utility into memory and executes the utility.
• After the execution is complete, the shell once again displays the prompt, conveying that it is ready for the next command.

This project involved understanding and further modifying the existing implementation of Lexical analyzer and YAAC parser to implement a custom interpreter on UNIX system. This work enables a user to specify customized "Patterns" to generate various "Tokens" through Lexical Analyzer.

Further these "Tokens" are fed to customized implementation of "YAAC" parser that enables a user to specify "Grammar"  to suit the requirements of custom interpreter.