Relating neural recordings to sensory representations
Neural decoding asks which aspects of a stimulus or experience can be inferred from measured brain activity. We develop learning methods that relate neural recordings to representations of images, language, speech, music and other sounds. These measurements differ substantially in their temporal resolution, spatial specificity and noise structure. Research therefore examines how useful representations can be learned within a recording modality and how information can be aligned across modalities such as EEG, MEG and fMRI.
Encoding and decoding provide complementary perspectives on this problem. Encoding models predict recorded responses from stimulus representations, while decoding models recover information about the stimulus from those responses. Contrastive learning and multimodal alignment provide ways to compare representations across these domains. Studies of visual reconstruction and retrieval examine both the identification of a matching stimulus and the generation of an image consistent with measured neural information.
Generative models, transfer and interpretation
Generative and foundation models supply representations that can be connected to neural data. Our work investigates latent diffusion methods, shared neural representations and architectures for transferring information across participants and datasets. Language and speech decoding extend these questions to neural and peripheral electrophysiological recordings. Temporal integration is an important part of this work because the information carried by a signal depends on when and over what interval it is measured.
Evaluation must distinguish information supported by the recording from structure introduced by the model. We examine generalisation across datasets and participants, the contribution of model complexity and alignment, and the interpretation of learned representations. Graph and geometric learning provide additional tools for describing relationships within complex data. Together, these methods support a quantitative account of what a neural decoding system learns, which distinctions it can recover and how consistently it operates beyond its training data.
Research resources
Profiles by research interest
Additional profiles by research interest (3)
Publications
2026
Neuroinformatics
Recent advancements in neural decoding have shown promising results in reconstructing visual experiences from brain activity. However, existing approaches focus primarily on decoding within a single dataset or subject, which limits generalization across various sources of neuroimaging. In this work, we propose a novel framework for the decoding of visual sti…Read the full abstractCollapse abstract
Recent advancements in neural decoding have shown promising results in reconstructing visual experiences from brain activity. However, existing approaches focus primarily on decoding within a single dataset or subject, which limits generalization across various sources of neuroimaging. In this work, we propose a novel framework for the decoding of visual stimuli between subjects and between data sets, integrating neural recordings from multiple publicly available fMRI datasets. To address inherent intersubject and interdataset variability, we introduce a contrastive learning-based alignment strategy using image embeddings from a pre-trained IP-Adapter model. Our approach learns a shared latent space by aligning subject-specific neural representations with image features, enabling generalized decoding across both subjects and datasets. In addition, we propose a simple yet effective data augmentation method using ridge regression. This method synthesizes realistic fMRI-like signals from novel images by predicting voxel activity and injecting learned noise distributions, thus enhancing training diversity and model robustness. To the best of our knowledge, while several recent studies have explored cross-subject decoding, we extend recent cross-subject decoding efforts by training a single unified framework jointly across multiple public fMRI datasets and subjects, enabling cross-dataset transfer in addition to cross-subject generalization. We distinguish this multi-dataset unified training setting, where each dataset contributes training data, from a stricter leave-one-dataset-out transfer setting in which the target dataset is excluded from source pretraining and used only for lightweight alignment-layer adaptation. Empirically, our unified model achieves strong semantic reconstruction across datasets (e.g., up to 94.8% CLIP similarity on NSD (AUG) and 0.403 SSIM on BOLD5000 after lightweight finetuning), demonstrating robust cross-subject and cross-dataset transfer.
2026
Imaging neuroscience (Cambridge, Mass.)
Characterizing the information content of intracortical signals during visual processing is a central challenge in systems neuroscience. We address the problem of decoding visual information from high-density intracortical recordings in primates, using the THINGS Ventral Stream Spiking Dataset. We systematically evaluate the effects of model architecture, tr…Read the full abstractCollapse abstract
Characterizing the information content of intracortical signals during visual processing is a central challenge in systems neuroscience. We address the problem of decoding visual information from high-density intracortical recordings in primates, using the THINGS Ventral Stream Spiking Dataset. We systematically evaluate the effects of model architecture, training objectives, and data scaling on decoding performance. Results show that decoding accuracy is jointly driven by non-linearity and selective temporal aggregation, rather than heavier sequence modelling in this data regime. A simple model combining temporal attention with a shallow MLP achieves up to 70% top-1 image retrieval accuracy, outperforming linear baselines as well as recurrent and convolutional approaches. Scaling analyses reveal predictable diminishing returns with increasing input dimensionality and dataset size. Building on these findings, we design a modular generative decoding pipeline that combines low-resolution latent reconstruction with semantically conditioned diffusion, generating plausible images from 200 ms of brain activity. This framework provides principles for brain-computer interfaces and semantic neural decoding.
2026
arXiv
A prominent theory in cognitive science suggests that concepts in the brain are organized as high-dimensional vectors, with semantic meaning captured by directions and relative angles in this space. Brain decoding is the effort of reconstructing or retrieving stimuli (or their representations) from neural activity and involves finding a function that approxi…Read the full abstractCollapse abstract
A prominent theory in cognitive science suggests that concepts in the brain are organized as high-dimensional vectors, with semantic meaning captured by directions and relative angles in this space. Brain decoding is the effort of reconstructing or retrieving stimuli (or their representations) from neural activity and involves finding a function that approximates how the brain represents concepts. This motivates the investigation of contrastive objectives as biologically plausible candidates to reverse the brain loss function. In this work, we study how functional MRI (fMRI) activity can generally be mapped with the embedding spaces of foundation models in vision, language, and audio. Although neural computations are highly non-linear at the microscale, fMRI measurements average signals across space and time, further smoothed by noise, effectively linearizing the observable representation. Consistent with these views, our experiments across multiple datasets demonstrate that linear contrastive decoders consistently outperform ridge regression and standard non-linear alternatives, and that these results generalize across images, text, and sound. These findings indicate that decoding gains arise more from the choice of training objective than from architectural complexity, pointing to contrastive-linear models as a principled strategy for brain decoding.
2026
arXiv
Understanding how speech foundation models relate to human cortical activity is a key challenge for computational neuroscience. Here, we investigate how internal representations from Whisper predict intracranial ECoG responses during naturalistic speech perception. We introduce a time-resolved neural encoder that combines speech embeddings with a recurrent t…Read the full abstractCollapse abstract
Understanding how speech foundation models relate to human cortical activity is a key challenge for computational neuroscience. Here, we investigate how internal representations from Whisper predict intracranial ECoG responses during naturalistic speech perception. We introduce a time-resolved neural encoder that combines speech embeddings with a recurrent temporal model and soft attention, allowing us to examine layer-wise brain alignment. Intermediate Whisper layers provide the strongest correspondence with neural activity, supporting a hierarchical match between model representations and cortical speech processing. Comparisons with baselines show that high-resolution ECoG responses benefit from temporally structured modelling beyond linear mappings from the same speech representations. In addition, attention maps reveal temporally local alignment between speech embeddings and neural responses, while a phonemic interpretability analysis identifies anatomically coherent phoneme-category organization among encoding-informative electrodes. Together, these results suggest that speech foundation models offer a useful framework for studying time-resolved cortical speech representations.
2026
arXiv
Recent progress in visual brain decoding from fMRI has been enabled by large-scale datasets such as the Natural Scenes Dataset (NSD) and powerful diffusion-based generative models. While current pipelines are primarily optimized for perception, their performance under mental-imagery remains less well understood. In this work, we study how a state-of-the-art…Read the full abstractCollapse abstract
Recent progress in visual brain decoding from fMRI has been enabled by large-scale datasets such as the Natural Scenes Dataset (NSD) and powerful diffusion-based generative models. While current pipelines are primarily optimized for perception, their performance under mental-imagery remains less well understood. In this work, we study how a state-of-the-art (SOTA) perception decoder (DynaDiff) can be adapted to reconstruct imagined content from the NSD-Imagery benchmark. We propose a latent functional alignment (LFA) approach that maps imagery-evoked activity to the pretrained model's semantic content-enriched conditioning space, by adding a simple alignment module, while keeping the original remaining components frozen. To mitigate the limited amount of matched imagery-perception supervision, we further introduce a neural retrieval-based augmentation strategy that selects semantically related NSD perception trials from the same participants. Across four subjects, LFA consistently improves high-level semantic reconstruction metrics relative to the frozen pretrained baseline and a voxel-space ridge alignment baseline, and enables above-chance decoding from multiple cortical regions. These results suggest that semantic structure learned from perception can be leveraged to stabilize and improve visual imagery decoding under out-of-distribution conditions.
Additional publications (9)
2026
Information Fusion
This paper presents a novel approach towards creating a foundational model for aligning neural data and visual stimuli across multimodal representationsof brain activity by leveraging contrastive learning. We used electroencephalography (EEG), magnetoencephalography (MEG), and functional magnetic resonance imaging (fMRI) data. Our framework's capabilities ar…Read the full abstractCollapse abstract
This paper presents a novel approach towards creating a foundational model for aligning neural data and visual stimuli across multimodal representationsof brain activity by leveraging contrastive learning. We used electroencephalography (EEG), magnetoencephalography (MEG), and functional magnetic resonance imaging (fMRI) data. Our framework's capabilities are demonstrated through three key experiments: decoding visual information from neural data, encoding images into neural representations, and converting between neural modalities. The results highlight the model's ability to accurately capture semantic information across different brain imaging techniques, illustrating its potential in decoding, encoding, and modality conversion tasks.
2025
Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference
Brain metastases (BM), along with primary central nervous system lymphomas and glioblastomas, represent the majority of malignant brain tumors encountered in clinical neuro-oncology, driving a need for advanced imaging techniques and post-processing methods to improve their characterization and treatment monitoring. In particular, stereotactic radiosurgery (…Read the full abstractCollapse abstract
Brain metastases (BM), along with primary central nervous system lymphomas and glioblastomas, represent the majority of malignant brain tumors encountered in clinical neuro-oncology, driving a need for advanced imaging techniques and post-processing methods to improve their characterization and treatment monitoring. In particular, stereotactic radiosurgery (SRS), a cornerstone treatment for BM, delivers high-dose, focused radiation (>20 Gy) to target lesions with minimal impact on surrounding tissues. Despite its efficacy, radiation-induced effects such as early radiation effects (ERE) and adverse radiation effects (ARE) complicate diagnosis and management, with ARE occurring in up to 30% of patients, often presenting as ring-enhancing T2/FLAIR hyperintensities. To address these challenges, we aimed to compare standard radiomics-based machine learning approaches with pretrained generative models for assessing ERE in BM lesions. A cohort of 21 patients for a total of 35 lesions (17 treatment-naïve and 18 post-SRS +/- combination therapy) who underwent multiparametric 18F-FPIA PET/MRI was analyzed. The study investigated: 1) Multiparametric analysis of PET and MRI diffusion/perfusion parameters (ADC, Ktrans, CBF, K1, vt); 2) MRI-based radiomics; 3) static PET radiomics; 4) Dynomics; 5) a combination of PET and MRI radiomics; and 6) low-level embeddings from a pretrained generative diffusion model applied to full T1, static PET, and their combination. Using manually contoured lesion masks for analyses 1-5 and lesion-free embeddings for analysis 6, multiple classifiers (SVM, XGBoost, Linear regressor) were applied after feature standardization and principal component analysis (retaining 90% variance). Fivefold cross-validation demonstrated comparable performances across radiomic approaches (Accuracy: 71.95±0.05%, AUC: 0.72±0.05%), while the pretrained generative model achieved significantly higher performance (Accuracy: 83.82±0.01%, AUC: 0.83±0.01%) without requiring lesion segmentation in assessing ERE in BM lesions.Clinical Relevance-This study shows the potential of generative models to streamline and enhance the assessment of early radiation effects in parenchymal metastatic lesions without need of lesion segmentation.
2025
Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference
Decoding visual stimuli from neural activity poses significant challenges due to the complexity of cross-subject neural variability and the hierarchical nature of visual processing. This study introduces a novel cross-subject brain decoding framework that integrates structural and semantic information to reconstruct images from fMRI data. Using diffusion mod…Read the full abstractCollapse abstract
Decoding visual stimuli from neural activity poses significant challenges due to the complexity of cross-subject neural variability and the hierarchical nature of visual processing. This study introduces a novel cross-subject brain decoding framework that integrates structural and semantic information to reconstruct images from fMRI data. Using diffusion models, we align neural representations with visual and textual embeddings through a contrastive learning paradigm. Our framework employs a composite neural module to harmonize cross-subject fMRI signals into a unified latent space, while a dual-pathway architecture, combining VDVAE for structural reconstruction and IP-Adapter with BERT for semantic alignment, ensures hierarchical fidelity. Evaluated on the Natural Scenes Dataset (NSD), our method achieves state-of-the-art performance in structural (SSIM: 0.379) and semantic (EffNet-B: 0.571 and SwAV: 0.225) metrics, outperforming previous studies. The results demonstrate robust generalizability across four subjects, advancing the feasibility of cross-subject brain decoding and providing insights into distributed neural encoding mechanisms.Clinical relevance-This study advances foundational understanding of neural encoding by jointly modeling structural and semantic information derived from brain activity, a novel methodological contribution to brain decoding research. The integration of these complementary dimensions provides a more comprehensive framework for interpreting how the brain hierarchically processes visual stimuli, offering insight into neural representation mechanisms.
2025
Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference
Brain decoding aims to reconstruct external stimuli from brain activity, providing insights into the neural representation of cognitive experiences. Music decoding from functional magnetic resonance imaging (fMRI) is particularly challenging due to the complexity of auditory processing and the temporal limitations of fMRI signals. In this study, we introduce…Read the full abstractCollapse abstract
Brain decoding aims to reconstruct external stimuli from brain activity, providing insights into the neural representation of cognitive experiences. Music decoding from functional magnetic resonance imaging (fMRI) is particularly challenging due to the complexity of auditory processing and the temporal limitations of fMRI signals. In this study, we introduce a novel decoding framework that improves the alignment between fMRI activity and latent musical representations extracted using a pre-trained multimodal model (CLAP). We propose a dual-loss approach combining Optimal Transport and Contrastive Learning to enhance feature mapping and retrieval accuracy. The first loss ensures structural consistency between brain-predicted and true musical embeddings, while the contrastive loss refines the embedding space by maximizing similarities between corresponding pairs and minimizing non-correspondences. Using fMRI data from five subjects listening to music tracks from the GTZAN dataset, our method achieves improved decoding performance, surpassing traditional regression-based approaches from 22.1% top-1 accuracy to 29.3%. These results highlight the potential of integrating Optimal Transport and Contrastive Learning to improve brain decoding performance, paving the way for extending the approach to different sensory domains and applications in Brain-Computer Interfaces (BCI).Clinical relevance- This study could have clinical implications for understanding auditory processing disorders and developing neurorehabilitation strategies. By elucidating how the brain encodes complex auditory stimuli, this approach may contribute to BCI applications for speech and music perception restoration in individuals with hearing impairments or neurological conditions affecting auditory cognition.
2025
Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE Engineering in Medicine and Biology Society. Annual International Conference
This study proposes an automated approach to radiotherapy treatment planning by integrating a reinforcement-learning-style iterative framework with a multimodal Large Language Model (LLM). We specifically investigate the problem of Beam Angle Optimization, a high-dimensional and non-convex subproblem of Treatment Planning. Our system employs GPT-4V to select…Read the full abstractCollapse abstract
This study proposes an automated approach to radiotherapy treatment planning by integrating a reinforcement-learning-style iterative framework with a multimodal Large Language Model (LLM). We specifically investigate the problem of Beam Angle Optimization, a high-dimensional and non-convex subproblem of Treatment Planning. Our system employs GPT-4V to select candidate beam angles and analyze three-dimensional dose distributions generated by Monte Carlo simulations within the MatRAD environment. Iterative plan refinement is guided by a reward function that encourages target dose conformity and penalizes excessive dose to organs at risk. We incorporate exploration-exploitation principles to strike a balance between investigating diverse action proposals and refining promising solutions. Experimental results on prostate cancer cases demonstrate that our LLM-based framework offers superior performance compared to random beam selection and can outperform the quality of deep reinforcement learning baselines, indicating the potential for LLMs to assist in complex radiotherapy treatment planning tasks.Clinical relevance-This approach is designed to alleviate the significant effort of manual treatment planning by assisting medical physicists in exploring beam configurations and systematically refining plans to improve dose coverage and protect healthy tissues.
2024
Computers in biology and medicine
Decoding visual representations from human brain activity has emerged as a thriving research domain, particularly in the context of brain-computer interfaces. Our study presents an innovative method that employs knowledge distillation to train an EEG classifier and reconstruct images from the ImageNet and THINGS-EEG 2 datasets using only electroencephalograp…Read the full abstractCollapse abstract
Decoding visual representations from human brain activity has emerged as a thriving research domain, particularly in the context of brain-computer interfaces. Our study presents an innovative method that employs knowledge distillation to train an EEG classifier and reconstruct images from the ImageNet and THINGS-EEG 2 datasets using only electroencephalography (EEG) data from participants who have viewed the images themselves (i.e. "brain decoding"). We analyzed EEG recordings from 6 participants for the ImageNet dataset and 10 for the THINGS-EEG 2 dataset, exposed to images spanning unique semantic categories. These EEG readings were converted into spectrograms, which were then used to train a convolutional neural network (CNN), integrated with a knowledge distillation procedure based on a pre-trained Contrastive Language-Image Pre-Training (CLIP)-based image classification teacher network. This strategy allowed our model to attain a top-5 accuracy of 87%, significantly outperforming a standard CNN and various RNN-based benchmarks. Additionally, we incorporated an image reconstruction mechanism based on pre-trained latent diffusion models, which allowed us to generate an estimate of the images that had elicited EEG activity. Therefore, our architecture not only decodes images from neural activity but also offers a credible image reconstruction from EEG only, paving the way for, e.g., swift, individualized feedback experiments.
2024
Journal of neural engineering
Objective.Brain decoding is a field of computational neuroscience that aims to infer mental states or internal representations of perceptual inputs from measurable brain activity. This study proposes a novel approach to brain decoding that relies on semantic and contextual similarity.Approach.We use several functional magnetic resonance imaging (fMRI) datase…Read the full abstractCollapse abstract
Objective.Brain decoding is a field of computational neuroscience that aims to infer mental states or internal representations of perceptual inputs from measurable brain activity. This study proposes a novel approach to brain decoding that relies on semantic and contextual similarity.Approach.We use several functional magnetic resonance imaging (fMRI) datasets of natural images as stimuli and create a deep learning decoding pipeline inspired by the bottom-up and top-down processes in human vision. Our pipeline includes a linear brain-to-feature model that maps fMRI activity to semantic visual stimuli features. We assume that the brain projects visual information onto a space that is homeomorphic to the latent space of last layer of a pretrained neural network, which summarizes and highlights similarities and differences between concepts. These features are categorized in the latent space using a nearest-neighbor strategy, and the results are used to retrieve images or condition a generative latent diffusion model to create novel images.Main results.We demonstrate semantic classification and image retrieval on three different fMRI datasets: Generic Object Decoding (vision perception and imagination), BOLD5000, and NSD. In all cases, a simple mapping between fMRI and a deep semantic representation of the visual stimulus resulted in meaningful classification and retrieved or generated images. We assessed quality using quantitative metrics and a human evaluation experiment that reproduces the multiplicity of conscious and unconscious criteria that humans use to evaluate image similarity. Our method achieved correct evaluation in over 80% of the test set.Significance.Our study proposes a novel approach to brain decoding that relies on semantic and contextual similarity. The results demonstrate that measurable neural correlates can be linearly mapped onto the latent space of a neural network to synthesize images that match the original content. These findings have implications for both cognitive neuroscience and artificial intelligence.
2023
arXiv
Recently, it has become progressively more evident that classic diagnostic labels are unable to reliably describe the complexity and variability of several clinical phenotypes. This is particularly true for a broad range of neuropsychiatric illnesses (e.g., depression, anxiety disorders, behavioral phenotypes). Patient heterogeneity can be better described b…Read the full abstractCollapse abstract
Recently, it has become progressively more evident that classic diagnostic labels are unable to reliably describe the complexity and variability of several clinical phenotypes. This is particularly true for a broad range of neuropsychiatric illnesses (e.g., depression, anxiety disorders, behavioral phenotypes). Patient heterogeneity can be better described by grouping individuals into novel categories based on empirically derived sections of intersecting continua that span across and beyond traditional categorical borders. In this context, neuroimaging data carry a wealth of spatiotemporally resolved information about each patient's brain. However, they are usually heavily collapsed a priori through procedures which are not learned as part of model training, and consequently not optimized for the downstream prediction task. This is because every individual participant usually comes with multiple whole-brain 3D imaging modalities often accompanied by a deep genotypic and phenotypic characterization, hence posing formidable computational challenges. In this paper we design a deep learning architecture based on generative models rooted in a modular approach and separable convolutional blocks to a) fuse multiple 3D neuroimaging modalities on a voxel-wise level, b) convert them into informative latent embeddings through heavy dimensionality reduction, c) maintain good generalizability and minimal information loss. As proof of concept, we test our architecture on the well characterized Human Connectome Project database demonstrating that our latent embeddings can be clustered into easily separable subject strata which, in turn, map to different phenotypical information which was not included in the embedding creation process. This may be of aid in predicting disease evolution as well as drug response, hence supporting mechanistic disease understanding and empowering clinical trials.
2022
arXiv
The lack of large labeled medical imaging datasets, along with significant inter-individual variability compared to clinically established disease classes, poses significant challenges in exploiting medical imaging information in a precision medicine paradigm, where in principle dense patient-specific data can be employed to formulate individual predictions…Read the full abstractCollapse abstract
The lack of large labeled medical imaging datasets, along with significant inter-individual variability compared to clinically established disease classes, poses significant challenges in exploiting medical imaging information in a precision medicine paradigm, where in principle dense patient-specific data can be employed to formulate individual predictions and/or stratify patients into finer-grained groups which may follow more homogeneous trajectories and therefore empower clinical trials. In order to efficiently explore the effective degrees of freedom underlying variability in medical images in an unsupervised manner, in this work we propose an unsupervised autoencoder framework which is augmented with a contrastive loss to encourage high separability in the latent space. The model is validated on (medical) benchmark datasets. As cluster labels are assigned to each example according to cluster assignments, we compare performance with a supervised transfer learning baseline. Our method achieves similar performance to the supervised architecture, indicating that separation in the latent space reproduces expert medical observer-assigned labels. The proposed method could be beneficial for patient stratification, exploring new subdivisions of larger classes or pathological continua or, due to its sampling abilities in a variation setting, data augmentation in medical image processing.