SHADOWSENSE
SENSING SOCIAL TOUCHES
ShadowSense is a low-tech, low-cost tactile sensing method to recognize touch gestures and touch positions on social robots for improving the robot’s awareness and promoting intelligent interaction behaviors. A camera inside a robot’s body captures the shadows created by touching its skin or even hovering over the skin, tracking touch positions, and detecting touch gestures.
The core interaction capabilities of ShadowSense include detecting touch activity, classifying touch gestures, and identifying the position of the touch event.
ShadowSense is able to detect even a light touch, which applies minimal force. Besides direct physical contact, over-the-skin input can be detected as well. ShadowSense supports the classification of touch gestures when an activity is detected. This allows for analyzing the social meanings or user intentions behind a touch gesture.
ShadowSense also allows for tracking the position of the shadow, further enriching the interaction possibilities. , It can perform over-the-skin shadow tracking, for example, tracking the position of human bodies when they stand close to the skin, anticipating touch activities.
We use a Densely Connected Convolutional Neural Network to recognize touch gestures from shadow images. In an experiment, we evaluated six interaction gestures: a palm touch, punch, two hands, hug, point, and nothing. The classifier was evaluated with three lighting conditions: daylight, dusk, and night. The results showed high accuracy in gesture recognition, between 87.5% and 96.0%, depending on the lighting.
ShadowSense lowers the barrier to developing touch-enabled social robots or minimalist robotic devices. For example, within minutes, the balloon could integrate ShadowSense and become a touch-sensitive object, which can respond by illuminating an LED strip connected to an off-board Arduino.
I got inspiration from lights and shadows. I love how the shadow projects to the wall and shifts its shape with the sun moving throughout the day. I was wondering how much information could a shadow convey and whether we are able to access it.
This photo was taken by Lynn Johnson from National Geographic to recreate the story.
This project has been selected to include in the story "The power of touch" of National Geographic Magazine, June 2022.
Publication
Hu, Yuhan, Sara Maria Bejarano, and Guy Hoffman. “ShadowSense: Detecting Human Touch in a Social Robot Using Shadow Image Classification.” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT/UbiComp) 4.4 (2020): 1‑24.