Medical Physics,
AI and Neurotechnology

Music is a universal phenomenon that influences human experiences across cultures. We investigate whether music can be decoded from human brain activity measured with fMRI, by modeling mappings between neural data and latent representations of musical stimuli. Our approach integrates functional and anatomical alignment techniques to facilitate cross-subject decoding. Starting from the GTZan fMRI dataset, where five participants listened to 540 musical tracks from 10 genres, we used the CLAP model to extract latent representations of the musical stimuli and developed voxel-wise encoding models to identify brain regions responsive to these stimuli, by applying a threshold to the correlation between predicted and actual brain activity. Our decoding pipeline, primarily retrieval-based, employs a linear map to project back brain activity to the corresponding CLAP features. This enables us to retrieve the musical stimuli most similar to those that originated the fMRI data. Our results demonstrate state-of-the-art identification accuracy, outperforming existing approaches.

RATIONALE AND OBJECTIVES: Automated semantic classification of radiology reports can enhance clinical workflows and research, but often relies on language- and task-specific models, limiting scalability. This study aims to evaluate and improve the 0-shot baseline performance of open-weight large language models (LLMs) in the semantic classification of free-text magnetic resonance imaging (MRI) reports.

MATERIALS AND METHODS: A corpus of 1385 reports from a single neuroradiology center was used to assess four LLMs (GPT-OSS-20B, Magistral-Small-2509, Gemma-4-31B, Qwen3.5-35B-A3B) of comparable scale (∼20-35B parameters). The brain MRI reports were labeled as Pathological or Non-Pathological, with the Pathological ones further classified into five diagnostic categories (Neoformation, Vascular, Inflammation/Infection, Neurodegenerative, Other). The LLMs were evaluated using zero- and few-shot prompting with majority-vote aggregation. Quantized Low-Rank Adaptation (QLoRA) fine-tuning was then applied to the best-performing model across multiple configurations. Performance was assessed using Macro F1, Micro F1, Precision, Recall, and Exact Match Accuracy.

RESULTS: The binary classification task achieved a Macro F1 score of 0.84 on the best-performing Qwen3.5-35B-A3B model, while multilabel classification reached 0.81. Majority voting (MV) outperformed individual-set predictions. QLoRA fine-tuning yielded a Macro F1 score of 0.83 on the multilabel task; the improvement over the 0-shot baseline was statistically supported for Macro F1, Micro F1, Macro Precision, and Exact Match, whose bootstrap 95% confidence intervals excluded zero, but not for Macro Recall. Fine-tuning and few-shot prompting strategies are complementary, with neither consistently outperforming the other. Error analysis showed that the models struggled with borderline annotation cases and often missed secondary labels.

CONCLUSION: Open-weight LLMs demonstrate strong capability for neuroradiology report classification without extensive task-specific training. Fine-tuning further improves performance, supporting integration into scalable semantic radiology data pipelines. However, the present evidence comes from a single Italian center without external validation, and the fine-tuning gain was estimated on a small held-out test set, for which the Macro F1 improvement, though statistically significant, has a confidence interval with a lower bound close to zero. In this light, the approach is best positioned as an internal indexing and semantic-retrieval tool, supporting faster report retrieval, structured cohort identification, internal audit, and scalable phenotyping for neuroradiology research and data warehousing, as well as efficient active-learning strategies, with broader generalizability requiring additional external multicentre validation before clinical deployment.

Visual decoding from fMRI is typically siloed by participant and experiment, obscuring whether heterogeneous neural measurements can be organized within a common computational geometry. Here we introduce BRAID-fMRI (Brain Representation Alignment across Individuals and Datasets), a shared CLIP-supervised decoding framework. BRAID-fMRI uses a single ROI-wise Transformer with optional participant conditioning across eight visual-fMRI datasets comprising 93 dataset-specific participant entries, 430,007 single-trial responses and 162,839 unique stimuli. Regional brain activity is aligned with 512-dimensional CLIP ViT-B/32 representations using a multi-positive contrastive objective that treats repeated stimuli across participants and datasets as positives. BRAID-fMRI supports retrieval across seven evaluation datasets. On eight matched participant entries, it achieves 35.0 +/- 11.1% Top-10 accuracy, exceeding the observed mean accuracy of the two evaluated baselines – the MindEye-style pooled-CLIP decoder (27.1 +/- 5.1%) and ridge regression (21.1 +/- 9.1%) – and attaining the highest observed accuracy for seven of eight entries. In separately trained participant-agnostic models, expanding the source pool increased target-dataset holdout accuracy by up to 92.1% relative to the initial source-training condition. The learned space preserves graded semantic structure, while ablations and saliency highlight ventral and early visual cortex and category-specific motion and attentional systems. These results support scalable cross-dataset decoding into a common CLIP-aligned space, with model sensitivity concentrated in ventral and early visual inputs.