iShadow: The Computational Eyeglass System

Overview

The human eye offers a fascinating window into an individual's personality traits, medical problems, brain abnormalities, behavioral conditions, cognitive attention, and decision making. These characteristics have made it the subject of decades of research by experts in cognition, ophthalmology, neuroscience, epidemiology, behavior, and psychiatry, who have not only enhanced our understanding of how the eye works, but also revealed new ways of diagnosing health concerns at an early stage. Unobtrusive and continuous monitoring of these signals can provide significant benefits in safety and health of patients and professionals. In addition, a device that enables continuous eye movement monitoring in field settings can provide information regarding cognitive processes and fatigue.

However, integrating an eye tracker into a pair of spectacles is remarkably challenging. A real-time eye tracker involves multiple eye-facing imagers sampling at frame rates of tens of Hz (often upwards of 100Hz) thereby generating megabits of data per second, has complex image processing requirements to extract various parameters of the eye, and has substantial communication needs for transmitting this information to a device such as a smartphone. As a consequence, state-of-art eye trackers like SMI and Tobii Glass are designed for short-term data steaming or storage, and are tethered to a portable smartphone-sized accessory with batteries, computation and storage.

iShadow is our effort to address this challenge and design a truly wearable, unobtrusive, ultra-low power eye tracker.

Drowsiness and Fatigue sensing using iLid: Ubicomp 2017

Pupil dilation and Saccade monitoring @ 200Hz: presented at Mobicom 2015

Sparse sampling for low-power eye tracking using iShadow (version 1): presented at MobiSys 2014

Publications

iLid: Low-power Sensing of Fatigue and Drowsiness Measures on a Computational Eyeglass
Soha Rostaminia, Addison Mayberry, Deepak Ganesan, Benjamin Marlin, Jeremy Gummeson
Proceedings of ACM Ubicomp 2017.

CIDER: Enabling Robustness-Power Tradeoffs on a Computational Eyeglass
Addison Mayberry, Yamin Tun, Pan Hu, Duncan Smith-Freedman, Benjamin Marlin, Christopher Salthouse, Deepak Ganesan
Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (MobiCom '15)

iShadow: Design of a Wearable, Real-time Mobile Gaze Tracker
Addison Mayberry, Pan Hu, Benjamin Marlin, Christopher Salthouse, Deepak Ganesan
Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '14)

iShadow: The Computational Eyeglass System (Best Demo Award)
Addison Mayberry, Pan Hu, Benjamin Marlin, Christopher Salthouse, Deepak Ganesan
Proceedings of the Symposium on Eye Tracking Research and Applications (ETRA '14)

Hamid Dadkhahi and Marco F. Duarte, “Masking Strategies for Image Manifolds,” submitted for publication, 2015.

Hamid Dadkhahi, Marco F. Duarte, and Benjamin Marlin, “Isomap Out-of-Sample Extension for Noisy Time Series Data,” IEEE International Workshop on Machine Learning in Signal Processing (MLSP), 2015, Boston, MA.

Hamid Dadkhahi and Marco F. Duarte, “Image Masking Schemes for Local Manifold Learning Methods,” IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2015, Brisbane, Australia, pp. 5768-5772.

Hamid Dadkhahi and Marco F. Duarte, “Masking Schemes for Image Manifolds,” IEEE Statistical Signal Processing Workshop (SSP), 2014, Gold Coast, Australia pp. 256-259. (Poster)

People

Many other people have contributed to the design, implementation and algorithms in iShadow. Special thanks to Geoff Barrows (CentEye), Russ Bielawski who designed the first generation of iShadow glasses.

Funding

This material is based upon work supported by the NSF under awards #1551850, #1217606, #1218586, and #1239341, as well as the NIH under grant #1U54EB020404. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF or the NIH.