@fraunhofer-ims/aifes-gesture-recognition

0
0
flwr new @fraunhofer-ims/aifes-gesture-recognition

Note: The source code will be published soon.

AIfES x Flower - Gesture Recognition with Federated Learning

This example demonstrates federated learning on embedded devices using AIfES (Artificial Intelligence for Embedded Systems) and Flower. The application performs gesture recognition directly on the M5StickC Plus 2 microcontroller using IMU sensor data and synchronizes the trained model with a federated learning server via MQTT.

Project Overview

This demo trains a small neural network directly on the device, synchronizes the local model with a server using federated learning, and then classifies gestures in real-time. The system can recognize three different gestures after recording only 5 training samples per class.

Key Features

  • On-device Training: Train a neural network directly on the microcontroller with AIfES
  • Federated Learning: Synchronize local models with a global model using Flower and MQTT
  • Automatic Gesture Detection: IMU-based motion detection without button input
  • Real-time Classification: Classify gestures on the device after federated synchronization

Set up the project

Prerequisites

Hardware:

  • M5StickC Plus 2 microcontroller

Software:

  • Arduino IDE (version 1.8.x or 2.x)
  • Python 3.8+ (for the federated learning server)

Fetch the app

Install Flower and required Python dependencies:

pip install flwr torch paho-mqtt numpy

Clone or download this repository:

git clone https://flower.ai/apps/fraunhofer-ims/aifes-gesture-recognition/
cd aifes-gesture-recognition

Project Structure

aifes-gesture-recognition
├── 01_Flower_AIfES_MQTT.ino    # Main firmware for M5StickC Plus 2
├── aifes_f32_fnn.h              # Neural network configuration
├── fl_comm_mqtt.h               # MQTT communication layer
├── fl_server_mqtt.py            # Federated learning server
├── pyproject.toml               # Python project dependencies (optional)
└── README.md

Install dependencies

Arduino Dependencies

  1. Install the Arduino IDE from arduino.cc/en/software
  2. Add the M5StickC Plus 2 board support via Board Manager
  3. Install the following libraries via Library Manager:

Configure WiFi and MQTT

Open fl_comm_mqtt.h and configure your WiFi credentials:

#define FL_WIFI_SSID     "YourSSID"
#define FL_WIFI_PASSWORD "YourPassword"

Optionally, configure the MQTT broker (default: broker.hivemq.com):

#define FL_MQTT_BROKER_HOST "broker.hivemq.com"
#define FL_MQTT_BROKER_PORT 1883

Run the Federated Learning Server

Start the federated learning server on your computer:

python fl_server_mqtt.py

The server will:

  • Connect to the MQTT broker
  • Publish the initial global model
  • Wait for client updates
  • Aggregate models using FedAvg when at least 2 clients have sent updates
  • Publish the updated global model

You should see output like:

Connecting to MQTT broker broker.hivemq.com:1883
Topic prefix: demo/fl
Connected to MQTT broker: reason_code=0

Run on the M5StickC Plus 2

Upload the Firmware

  1. Open 01_Flower_AIfES_MQTT.ino in Arduino IDE
  2. Select Tools → Board → M5StickC Plus 2
  3. Select the correct serial port under Tools → Port
  4. Click Upload to flash the firmware

Record Training Data

After uploading, the device will guide you through data collection:

  1. Hold the device in the same position throughout the process
  2. Perform Gesture 1 five times
    • Try to perform the gesture consistently
    • Wait briefly between repetitions
    • Ensure the motion exceeds the acceleration threshold
  3. Perform Gesture 2 five times
  4. Perform Gesture 3 five times

What gestures work well?

  • Arm flex
  • Quick punch motion
  • Twist or rotation
  • Sideways movement

Training and Synchronization

After data collection, the device will:

  1. Connect to WiFi
  2. Download the initial global model from the MQTT broker
  3. Train the local model with the recorded data
  4. Upload the local model to the federated learning server
  5. Wait for the aggregated global model
  6. Switch to classification mode

Classification Mode

Once in classification mode:

  • Perform one of your trained gestures
  • The display shows the predicted probabilities for all three classes
  • Example output: 
    Predicted gesture:
Gesture 1: 85%
Gesture 2: 10%
Gesture 3: 05%

Configuration Options

Model Architecture

You can modify the neural network architecture in aifes_f32_fnn.h:

#define DENSE_NEURONS_1 4    // Neurons in first hidden layer
#define DENSE_NEURONS_2 3    // Neurons in output layer (number of gestures)

Data Collection Parameters

Adjust these constants in 01_Flower_AIfES_MQTT.ino:

#define NUM_GESTURES 3           // Number of gesture classes
#define NUM_SAMPLES 20           // IMU samples per gesture
#define SAMPLES_PER_CLASS 5      // Training samples per class
#define ACCEL_THRESHOLD 4        // Motion detection threshold

Federated Learning Settings

Modify server settings in fl_server_mqtt.py:

MIN_CLIENTS = 2              # Minimum clients for aggregation
MQTT_BROKER_HOST = "..."     # MQTT broker address
MQTT_TOPIC_PREFIX = "demo/fl" # Topic namespace

Troubleshooting

WiFi connection fails:

  • Check SSID and password in fl_comm_mqtt.h
  • Ensure 2.4 GHz WiFi network (M5StickC Plus 2 doesn't support 5 GHz)

Model download timeout:

  • Ensure the Python server is running
  • Check MQTT broker connectivity
  • Verify the topic prefix matches between client and server

Gesture recording not starting:

  • Increase ACCEL_THRESHOLD if gestures are too subtle
  • Decrease ACCEL_THRESHOLD if motion starts recording unintentionally

Model aggregation not happening:

  • Ensure at least MIN_CLIENTS have sent updates
  • Check server logs for error messages

Advanced Features

Calibration

To remove gravity offset, uncomment the calibration in setup():

calibrate();

Secure MQTT Connection

For production use, configure TLS:

#define FL_MQTT_BROKER_PORT 8883
// Configure WiFiClientSecure instead of WiFiClient

Multiple Devices

To train with multiple M5StickC devices:

  1. Flash the firmware to multiple devices
  2. Ensure they all connect to the same MQTT broker
  3. The server will aggregate models from all clients

Resources

Community

License

This project is licensed under the GNU Affero General Public License v3.0 (AGPL-3.0).

Copyright (C) 2020-2026 Fraunhofer Institute for Microelectronic Circuits and Systems.

See the license headers in the source files for more details.

Citation

If you use this project in your research, please cite:

@misc{aifes_flower_gesture,
  title={AIfES x Flower: Federated Learning for Gesture Recognition on Embedded Devices},
  author={Fraunhofer Institute for Microelectronic Circuits and Systems},
  year={2026},
  howpublished={\url{https://flower.ai/apps/fraunhofer-ims/aifes-gesture-recognition/}}
}