NeuroSync: Generalized Brain Decoding of Visual Stimuli Across Subjects

Decoding visual stimuli from neural activity poses significant challenges due to the complexity of cross-subject neural variability and the hierarchical nature of visual processing. This study introduces a novel cross-subject brain decoding framework that integrates structural and semantic information to reconstruct images from fMRI data. Using diffusion models, we align neural representations with visual and textual embeddings through a contrastive learning paradigm. Our framework employs a composite neural module to harmonize cross-subject fMRI signals into a unified latent space, while a dual-pathway architecture, combining VDVAE for structural reconstruction and IP-Adapter with BERT for semantic alignment, ensures hierarchical fidelity. Evaluated on the Natural Scenes Dataset (NSD), our method achieves state-of-the-art performance in structural (SSIM: 0.379) and semantic (EffNet-B: 0.571 and SwAV: 0.225) metrics, outperforming previous studies. The results demonstrate robust generalizability across four subjects, advancing the feasibility of cross-subject brain decoding and providing insights into distributed neural encoding mechanisms.Clinical relevance-This study advances foundational understanding of neural encoding by jointly modeling structural and semantic information derived from brain activity, a novel methodological contribution to brain decoding research. The integration of these complementary dimensions provides a more comprehensive framework for interpreting how the brain hierarchically processes visual stimuli, offering insight into neural representation mechanisms.