Skip to content

Arduino/ESP32 3D gesture recognition with MPU6050 and DTW algorithm

Notifications You must be signed in to change notification settings

flooxo/MagicWand-Gesture-recognition

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

25 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

MagicWand-Gesture-recognition

Real working magic wand

Using an Arduino and a gyroscope/accelerometer as input for gesture recognition, commands can be executed in an almost magical way. It is encased in a 3D-printed housing that looks like Harry Potter's wand.

Functionality:

The accelerometer data is used in the Dynamic Time Warp algorithm to calculate the similarity compared to a previously recorded pattern. If the pattern matches (=below a certain threshold), an action is executed. In my case I wanted to control my LED strips, so I send an IR signal when the gesture is detected.
The serial output is not needed for functionality, but for debugging. It is set up to work with Putty/VSCode and has some nice colors to make debugging easier. (https://xdevs.com/guide/color_serial/)

image
accelerometer data of one axis; green = sample patter, red = wrong pattern, other = similar to sample pattern

How is a gesture recognized?
As a preparation, the acceleration data from a certain period are stored in an array, the so-called sample pattern. In a second array the current motion data are stored. Now a similarity is to be calculated from these two data. This works by means of Dynamic Time Warping (DTW). DTW tries to create a best possible match of the two time dependent sequences by means of matching, deletion and insertion of data points. For this there is a so-called cost matrix. This contains calculated values, which are then always connected with the smallest neighbor and thus ideally form a diagonal. The area between the calculated path and the diagonal can be used to determine the similarity. Under a certain threshold it is accepted as a gesture

image
cost matrix visualization; data from pic above
The green diagonal would be if the gesture would match at every point
    -> area = 0 => similarity = 100%.
The blue path is a gesture that is very similar to the sample pattern, which you can see because it is very close to the diagonal
    -> area ≈ 950 -> similarity = ~95%
The red path is the inverse of the sample pattern, which is also clearly visible because it deviates strongly from the diagonal
    -> area ≈ 8500 -> similarity = ~20%
In purple is the area that is calculated to get the similarity.

Components:

  • ESP32 (DevKitC V4)
  • (ATTiny85/Arduino Pro Micro)
  • MPU6050 gyroscope
  • IR diode
  • cables

image
connection sketch

Roadmap/ToDos:

  • Recognize simple gesture
  • Recognize multiple gestures
  • Local constraints in the DTW cost matrix to aboard the calculation if outside of range
  • Auto-recognition of gestures
  • Add 3D files
  • ArduinoProMicro/ATTiny85 compatible
  • Substract gyro out of acc data!!!!
  • Serial progress bar for VSCode/Putty (when reccording or calcualting)

Plans:

20220707_204939 20220707_205002 Prototommy (Prototype 2nd Gen)

image 3D case plan

Miscellaneous:

Actually it was planned not to use an ESP32 but originally an ATTiny85. However, this is for the development and debugging rather less suitable. The advantage would be that this is much smaller and can be operated longer with a battery. As a compromise I still have an Arduino Pro Micro lying around, maybe the time will find that I can use this, but the memory must be adapted, since this is much smaller than that of the ESP32.
Another idea that just came up, instead of having the Arduino calculate everything, maybe just use it as input and send the data to a more powerful controller like a Raspberry Pi. The Pi then calculates everything and may even be able to do more advanced things due to its higher processing power. The results of the calculation are then converted into an action and sent back to the Arduino, which then performs the given instruction.


all information without guarantee of correctness, is only my hobby on the side; for questions, suggestions or improvements please contact me

About

Arduino/ESP32 3D gesture recognition with MPU6050 and DTW algorithm

Topics

Resources

Stars

Watchers

Forks