Embodied Interaction
Exploring how gestures and body movement shape digital forms and spatial experiences
2025 & 2018 · Digital Futures Workshop · Harvard Graduate School of Design
MediaPipe · Grasshopper · Arduino · Kinect · Processing
I. OVERVIEW
This series explores how gestures and body movement can become inputs for computational design and interactive systems. The projects use hand tracking and motion sensing to control physical mechanisms, deform digital geometry, and generate projected graphics.
Each study connects sensing, computational mapping, and feedback. Movement is detected and translated into a response that participants can observe and influence through further action.
II. CHAPTER 1: From Hand-Craft to “Hand-Craft”
Translating gestures into movement
Developed during the Digital Futures 2025 workshop, this project explores interactive systems as an extension of Bernard Cache’s work on the objectile. This concept describes a family of possible forms generated through shared rules and variable parameters. The project extends this principle to bodily interaction, proposing that hand gestures could continuously influence a system’s form and behaviour.
A proposed technical pipeline connects hand tracking, parametric modelling, and motor control. A webcam and MediaPipe would capture hand landmarks, which would be reconstructed as virtual hands in Rhino and Grasshopper. Relationships between these virtual hands and digital geometry would translate gestures into parameter changes, with Arduino-controlled motors producing corresponding physical movements.
The design task therefore includes defining how the system responds to movement. By adjusting the relationships between gestures, geometry, and motor control, designers could explore different forms of interaction within the same computational framework.
III. CHAPTER 2: Hand-Sculpting Bubble
Shaping geometry through gesture
Developed as the second phase of Hand-Craft, Hand-Sculpting Bubble investigates the digital modelling stage as an interactive experience.
Tracked hand landmarks are reconstructed as virtual hands within Rhino. Their movement interacts with a deformable bubble model, allowing changes in hand position to produce continuous changes in its surface. Grasshopper and Kangaroo connect the tracked geometry with the bubble’s simulated behaviour.
The prototype creates a feedback loop between gesture and form: participants move their hands, observe the deformation, and adjust their movements. It explores how bodily action can become part of a modelling process, with geometry experienced through ongoing interaction.
Collaborators: Jing Chang, Yilun Jiang.
IV. CHAPTER 3: Breathing Constellation
From individual movement to a shared visual field
Breathing Constellation translates movement within an installation into an evolving projected network. A Kinect depth sensor captures visitors’ movement within a defined range, while Processing converts the incoming data into points, traces, and connecting lines.
As visitors move, new marks accumulate and overlap. When movement stops, the constellation disappears. The projection makes bodily activity visible through an abstract graphic that changes with the people occupying the space.
This project extends the investigation from individual hand gestures to movement within a shared environment, exploring how participation can shape a collective visual experience.
Collaborators: Jonathan Yueng, Jing Chang, Ailing Zhang, Veronica Smith.
V. REFLECTION
Together, these studies explore how computational systems can be shaped through bodily action. Their different outputs, motor movement, geometric deformation, and projected graphics, provide contexts for investigating the relationship between movement, mapping, and feedback.