Hand gesture classification using high-quality structured data such as videos, im-
ages, and hand skeletons is a well-explored problem in computer vision. Alterna-
tively, leveraging low-power, cost-effective bio-signals, e.g., surface electromyo-
graphy (sEMG), allows for continuous gesture prediction on wearable devices.
In this work, we aim to enhance EMG representation quality by aligning it with
embeddings obtained from structured, high-quality modalities that provide richer
semantic guidance, ultimately enabling zero-shot gesture generalization. Specif-
ically, we propose EMBridge, a cross-modal representation learning framework
that bridges the modality gap between EMG and pose. EMBridge learns high-
quality EMG representations by introducing a Querying Transformer (Q-Former),
a masked pose reconstruction loss, and a community-aware soft contrastive learn-
ing objective that aligns the relative geometry of the embedding spaces. We eval-
uate EMBridge on both in-distribution and unseen gesture classification tasks and
demonstrate consistent performance gains over all baselines. To the best of our
knowledge, EMBridge is the first cross-modal representation learning framework
to achieve zero-shot gesture classification from wearable EMG signals, showing
potential toward real-world gesture recognition on wearable devices.
- † University of Southern California
- ** Work done while at Apple