A0669
Title: Multi-modal cross-masked autoencoder for digital health measurements
Authors: Jingjing Zou - University of California, San Diego (United States) [presenting]
Abstract: The growing prevalence of digital health technologies has led to the generation of complex multi-modal data, such as physical activity measurements simultaneously collected from various sensors of mobile and wearable devices. These data hold immense potential for advancing health studies, but current methods predominantly rely on supervised learning, requiring extensive labeled datasets that are often expensive or impractical to obtain, especially in clinical studies. To address this limitation, a self-supervised learning framework is proposed, called multi-modal cross-masked autoencoder (MoCA), that leverages cross-modality masking and the transformer autoencoder architecture to utilize both temporal correlations within modalities and cross-modal correlations between data streams. Theoretical guarantees are also provided to support the effectiveness of the cross-modality masking scheme in MoCA. Comprehensive experiments and ablation studies demonstrate that the method outperforms existing approaches in both reconstruction and downstream tasks.
