Cross-Modality Adaptive Sensing and Perception

Note: Please view the desktop site in order to see images.

Hand gesture recognition (HGR) is a crucial technology for human–computer interaction, robotics, and multimedia systems. While multimodal sensing — for example, combining RGB and depth data — can improve recognition accuracy, it often leads to significantly higher power consumption, limiting deployment on resource-constrained devices. 

To address this issue, we introduce CM-ASAP, a cross-modality adaptive sensing and perception framework designed to balance accuracy and energy efficiency. Rather than keeping sensing fixed throughout inference, CM-ASAP dynamically adjusts sensing behavior and modality usage based on the evolving context of the gesture, activating additional sensors only when they are likely to improve recognition.

The framework is built on three key empirical insights:

1. Sensing dominates power consumption — often far exceeding the cost of inference computation.
2. The benefit of multimodal fusion depends on the gesture class — some gestures gain little from additional modalities.
3. Early frames provide strong predictive cues — enabling early class hypotheses that guide later sensing decisions.

Leveraging these observations, CM-ASAP incorporates two complementary strategies:

• Frame-Rate Adaptation (FRA): Reduces the frame capture rate over time to reduce total volume of data collected.
• Cross-Modality FRA (CM-FRA): Selectively activates or deactivates sensing modalities based on the expected accuracy gain inferred from early frames.

We evaluated this framework on the EgoGesture dataset, showing that CM-ASAP improves both efficiency and performance of the perception system, achieving up to 0.7% higher accuracy while reducing power consumption by more than 300 mW. These results highlight the potential of cross-modality adaptive sensing for practical, low-power gesture recognition on edge devices.

Publications

[1] S. Hor, M. El-Khamy, Y. Zhou, A. Arbabian and S. Lim, "CM-ASAP: Cross-Modality      Adaptive Sensing and Perception for Efficient Hand Gesture Recognition," 2024 IEEE 7th       International Conference on Multimedia Information Processing and Retrieval (MIPR), San Jose, CA, USA, 2024, pp. 207-213, doi: 10.1109/MIPR62202.2024.00039.