This is the all-in-one illustraion and troubleshooting for my PhD journey.

Outline

1. Raspberry Pi (Python) earphone

2. Arduino (C) earphone

3. EXG earphone

4. Multi-modal learning

5. 3D model

Raspberry Pi earphone

The below code has been tested on Rasbian11

1. check i2c status:

# Google search "Raspberry Pi, pinout"
i2cdetect -y 0/1 (two bus on RPi)

2. set the i2c speed:

# may need sudo apt update before
sudo nano /boot/config.txt
# add or revise the following lines
dtparam=i2c_arm=on
dtparam=i2c_arm_baudrate=1000000
dtparam=i2c0=on
dtparam=i2c0_baudrate=1000000

3. BMI160, BMI-270

4. bluetoothe HSP profile on Linux. I believe it only works for Headset, not True Wireless Stereo earphone. (don't try to record audio from AirPods or Galaxy Buds on Linux!)

Arduino earphone

1. Arduino BLE Sense Nano microphone recording Note that the data from serial should be converted to short (16-bit) + 8000 sample rate (HFS profile)

2. Arduino BLE Sense Nano Rev2: IMU is updated to BMI270 (finished, at arduino/arduino_code/bmi270.ino)

3. Serial data for two sensors: /arduino/receive.py (still working on it) next step: interrupt + FIFO to reach stable sample rate without loss

4. Arduino work on BMI323 (New IMU 2023), Mouser. You may need 1.27mm to 2.54mm adaptor to connect.

EXG

1. OpenBCI-earable

2. EMOTIV, IDUN ....

Multi-modal

I list all the multi-modal tasks worth exploring.

1. Camera+Point cloud, BEVFusion

2. Visual question answering (instruction comprehension), MobiVQA, efficient VLM, COSM2IC

3D model

1. insert-based IMU + commercial earphone

2. Full earphone

3. Bio-sensing earphone

4. out-eardrum earphone