Back to Projects

Project Overview

Developed for the Multisensory Interactive Systems course (Prof. Luca Turchet, 2024-2025), this project investigates whether haptic feedback and background music tempo influence the learning curve of the American Sign Language (ASL) alphabet.

We designed a smart glove equipped with flex sensors that detects hand gestures in real-time. This hardware interacts with a custom-built memory game, providing users with vibration feedback when they correctly form a letter.

Technical Implementation

The system relies on a seamless communication loop between hardware and software. The glove captures analog data, Arduino processes it, and Processing visualizes the game.

🔧 Hardware Components

  • Arduino Micro-controller
  • 5x Flex Sensors
  • Vibration Motor
  • 10Ω Resistors & Breadboard

💻 Software Stack

  • Arduino IDE 2.3.3 (Sensor Logic)
  • Processing 4.3 (Game UI & Logic)
  • Pure Data 0.54.1 (Sound Synthesis)

User Experience & Gameplay

The user engages in a "Memory-Learn" game designed to teach ASL letters through repetition and feedback.

1. Calibration

To account for different hand sizes and finger flexibility, the game begins with a calibration phase. The user holds their hand open for 3 seconds, then makes a tight fist for 3 seconds. This sets the min/max values for the flex sensors.

2. The Game Loop

The game consists of three levels. A letter is highlighted in purple on the screen, and the user must replicate the gesture. When the sensors detect the correct shape, the user must hold it for 3 seconds to "lock it in." The sequence is randomized to ensure true learning rather than pattern memorization.

3. Evaluation

Upon completion, the system displays the time taken for each level. We then test short-term retention by asking the user to perform gestures without visual aid.

System Setup

For those accessing the repository, the system requires the three component directories to run simultaneously:

  1. Upload Firmware: Flash the ArduinoSignLanguageGame.ino sketch to the connected microcontroller.
  2. Start Audio: Open MAIN.pd in Pure Data to handle the audio engine.
  3. Launch Game: Open ProcessingSignLanguageGame.pde and press Play to start the interface.