Search PubMed⌕ Search

SEARCH · Search PubMed

Results for “multimodal data”

Search indexed PubMed citations on genomics, clinical trials, systematic reviews and public health. Explore titles, authors and supplied subject terms, then open the PubMed record.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 55 records · Page 3Linked to original sources
PubMed1990

Image correlation in oncology.

Image correlation techniques can provide objective spatial registration between multimodality data sets acquired during the planning and follow-up phases of radiation therapy. Correlation of pre-CT/MRI with follow-up CT/MRI and 3D dose matrices may provide insights into normal tissue tolerance. Correlation of SPECT and planar scintigraphs with anatomical maps derived from CT/MRI may be useful in the precise localization of disease and in the evaluation of new modalities, such as radiolabeled monoclonal antibodies, in the diagnosis and treatment of cancer. Correlation of PET and MRI may lead to a more precise understanding of structure-function relationships of the brain. The development and refinement of multimodality image-correlation techniques is a logical step in the evolving role of imaging in radiation therapy.

Brain Neoplasms↗
PubMed2026

Biological Foundation Models for Complex Disease Research and Clinical Translation.

Complex diseases, including cancer, rare genetic disorders, neurodevelopmental and psychiatric conditions, and neurodegenerative diseases, arise from interactions among genetic variation, gene regulation, and cellular states that are difficult to capture using a single data type or biological scale. Biological foundation models address this challenge by treating nucleotides and genes as tokens and learning representations that can be transferred to downstream biomedical and clinical tasks. In this review, we examine two major model classes, genomic sequence foundation models and cell foundation models, and compare their tokenization strategies, model architectures, pretraining objectives, and adaptation methods. We summarize their emerging applications in regulatory variant interpretation, disease-associated cell-state analysis, drug-response prediction, and therapeutic target discovery across complex diseases. We distinguish applications supported by experimental or retrospective validation from those that remain primarily computational or conceptual. We further discuss key challenges to clinical translation, including multimodal data integration, model interpretability, benchmarking, patient-specific prediction, and privacy protection. We highlight future opportunities to integrate biological foundation models with emerging frameworks of medical digital twins, agentic AI, and federated learning. By linking model design to translational goals, this review provides a practical framework for evaluating biological foundation models and their readiness for complex disease research and clinical use.

biological foundation model↗
PubMed2025

Multimodal deep learning for immunotherapy response prediction and biomarker discovery in non-small cell lung cancer.

OBJECTIVE: Immunotherapy has emerged as a promising treatment for advanced non-small cell lung cancer (NSCLC), but accurately predicting which patients will benefit from it remains a major clinical challenge. To address this, we aim to develop a novel multimodal method, DeepAFM, that integrates histopathology, genomic features, and clinical information to predict patient responses to anti-PD-(L)1 immunotherapy. MATERIALS AND METHODS: A total of 93 patients with advanced NSCLC were included in this study. Histopathological whole-slide images were processed using a self-supervised VQVAE2 for representation learning. PCA and K-means clustering were then applied for dimensionality reduction and feature grouping. Key regions of interest were visualized through permutation importance evaluation and color-coding techniques. The extracted histopathological features, along with genomic alterations and clinical variables, were integrated into the DeepAFM multimodal prediction model. RESULTS: The DeepAFM achieved a high predictive performance with an area under the curve (AUC) of 0.77 (95% confidence interval: 0.69-1.00). Attention-based heatmaps revealed that the model could identify critical pathological patterns, genomic mutations, and clinical indicators associated with patient responses to immunotherapy. DISCUSSION: The integration of multimodal data enabled the model to capture complex interactions among pathology, genomics, and clinical characteristics, enhancing the interpretability and predictive power of immunotherapy response prediction. The visualization techniques facilitated the identification of biologically meaningful features and potential biomarkers. CONCLUSION: This study demonstrates the effectiveness of the DeepAFM in predicting responses to immunotherapy in advanced NSCLC. The approach not only improves prediction accuracy but also provides valuable insights for personalized treatment strategies and biomarker discovery.

Humans↗
PubMed2026

AI-Based 3D Heterogeneous Network Model for Functional Prediction of Epigenetics.

Human biology and diseases are the result of constantly evolving processes within an intricately complex molecular network of interactions, such as epigenetic regulation. Epigenetics refers to heritable changes in gene expression that occur without alterations to the underlying DNA sequence. These changes, driven by mechanisms such as DNA methylation, histone modifications, and noncoding RNAs, play critical roles in regulating chromatin structure and gene activity. Epigenetic regulation offers valuable insights into biological systems, and when integrated with sophisticated analyses, it enables us to gain insights into gene regulation and cellular behavior. Here, we describe an artificial intelligence (AI)-based model that is capable of generating 3-dimensional (3D) heterogeneous network by integrating multimodal data for the functional prediction of epigenetic mechanisms, emphasizing its applications in medicine, developmental biology, and personalized therapeutics. Heterogeneous networks in biology are powerful tools for understanding the complex interactions and interdependencies within biological systems. Key advancements in AI and multiomics data integration have propelled this field, offering new insights into disease mechanisms, biomarker discovery, and therapeutic interventions.

Epigenesis, Genetic↗
PubMed2000

Advances in neuroimaging of acute stroke.

As therapeutic options for treating acute stroke evolve, neuroimaging strategies are assuming an increasingly important role in the initial evaluation and management of patients. There is a recognized need for objective neuroimaging methods to identify the best candidates for early intervention. Both acute and long-term treatment decisions for stroke patients should optimally incorporate information provided by neuroimaging studies regarding tissue viability (eg, size, location, vascular distribution, degree of reversibility of ischemic injury, presence of hemorrhage), vessel status (site and severity of stenoses and occlusions), and cerebral perfusion (size, location, and severity of hypoperfusion). The ability to acutely identify the ischemic penumbra and to use this information to make treatment decisions may be within reach, particularly with the multimodal data provided by magnetic resonance techniques. This article will review recent developments in the field of neuroimaging of acute stroke and discuss the clinical applications of specific techniques of magnetic resonance imaging, computed tomography, positron emission tomography, single photon emission tomography, catheter angiography, and ultrasound imaging.

Cerebral Angiography↗
PubMed2026

Machine learning for population-level risk prediction of future cholangiocarcinoma.

BACKGROUND: The poor prognosis of cholangiocarcinoma (CCA) is largely driven by rapid, asymptomatic disease progression, which usually results in a late diagnosis in the absence of established screening strategies. An early, cost-effective, and universally applicable risk assessment strategy would therefore be valuable. METHODS: We developed machine learning (ML) models on prospective, multimodal data from 487,495 UK Biobank (UKB) participants, of whom 649 developed CCA during follow-up. Data from England (80%) were utilised for ML development via five-fold cross-validation, and then all models were tested on withheld data from Scotland, Wales, and Newcastle (20%). Iterative ablation studies reduced inputs from >150 features across demographic data, lifestyle, health records, blood parameters, genomics, and metabolomics to models built on five and ten routinely available clinical parameters. These were externally validated in the Penn Medicine Biobank (PMBB; n = 2638; 28 CCA), All of Us Research Program (AOU; n = 330,433; 362 CCA), Japan Medical Data Centre Claims Database (JMDC; n = 8,425,522; 723 CCA) and TriNetX (n = 728,886; 1592 CCA). FINDINGS: We show that ML models integrating biliary-disease associated health records and Gamma glutamyltransferase can stratify risk of future CCA. Evaluation on the UKB test set as well as three independent cohorts revealed robust performance and generalisability across ethnicities. We achieved AUROCs of 0.71 [95% CI: 0.703-0.711], 0.77 [95% CI: 0.764-0.778 ], 0.796 [95% CI: 0.795-0.798] and 0.8 [95% CI: 0.794-0.805] for UKB, PMBB, AOU, and JMDC respectively, with respective AUPRCs of 0.014 [95% CI: 0.009-0.018], 0.042 [95% CI: 0.037-0.048], 0.038 [95% CI: 0.033-0.042] and 0.001 [95% CI: 0.001-0.001]. In AOU, application of the Youden J-optimised threshold yielded a number needed to screen of 79. Separate models for intra- and extrahepatic CCA did not improve performance. In line with the pathophysiology, performance declined for longer intervals between assessment and event. A group-level analysis in the TriNetX cohort revealed hazard ratios of up to 82.5 [95% CI: 26.4-257.96]. We provide extensive interpretability results and release all source codes used to develop the presented models. INTERPRETATION: We provide a comprehensive framework for early CCA risk stratification in the general population, identifying key predictors, and demonstrating the potential of data-driven models in personalised screening for hepatobiliary cancer. FUNDING: German Cancer Aid (grant #70115730), Junior Principal Investigator Fellowship programme of RWTH Aachen Excellence strategy.

Humans↗
PubMed1999

MRI-SPECT fusion for the synthesis of high resolution 3D functional brain images: a preliminary study.

Medical imaging being a fast-expanding field, multimodal data fusion appears more and more as a key element for the optimal use of images. By fusion, we mean the combination of several information sources (in particular images), with the aim of providing either more condensed or more pertinent information. The long term scope of this work would be to improve the interpretation of 3D brain images, providing extra elements for the diagnosis and patient follow up. This preliminary study is part of a wider context: the medical follow up of patients suffering from probable Alzheimer disease observed in single photon emission tomography by fusion after registration with magnetic resonance images. Several information combination techniques based on the possibility theory are presented. A new operator, more specifically adapted to the fusion of anatomical and functional images, as well as a high resolution functional image synthesis technique are proposed. A first comparative study of fusion techniques is then proposed. Although no thorough test protocol has been defined, these preliminary results are encouraging, giving access to a wide field of potential clinical applications.

Aged↗
PubMedDate not supplied

A virtual reality patient simulation system for teaching emergency response skills to U.S. Navy medical providers.

Rapid and effective medical intervention in response to civil and military-related disasters is crucial for saving lives and limiting long-term disability. Inexperienced providers may suffer in performance when faced with limited supplies and the demands of stabilizing casualties not generally encountered in the comparatively resource-rich hospital setting. Head trauma and multiple injury cases are particularly complex to diagnose and treat, requiring the integration and processing of complex multimodal data. In this project, collaborators adapted and merged existing technologies to produce a flexible, modular patient simulation system with both three-dimensional virtual reality and two-dimensional flat screen user interfaces for teaching cognitive assessment and treatment skills. This experiential, problem-based training approach engages the user in a stress-filled, high fidelity world, providing multiple learning opportunities within a compressed period of time and without risk. The system simulates both the dynamic state of the patient and the results of user intervention, enabling trainees to watch the virtual patient deteriorate or stabilize as a result of their decision-making speed and accuracy. Systems can be deployed to the field enabling trainees to practice repeatedly until their skills are mastered and to maintain those skills once acquired. This paper describes the technologies and the process used to develop the trainers, the clinical algorithms, and the incorporation of teaching points. We also characterize aspects of the actual simulation exercise through the lens of the trainee.

Algorithms↗
PubMed2005

Double-wavelet approach to studying the modulation properties of nonstationary multimode dynamics.

On the basis of double-wavelet analysis, the paper proposes a method to study interactions in the form of frequency and amplitude modulation in nonstationary multimode data series. Special emphasis is given to the problem of quantifying the strength of modulation for a fast signal by a coexisting slower dynamics and to its physiological interpretation. Application of the approach is demonstrated for a number of model systems, including a model that generates chaotic dynamics. The approach is then applied to proximal tubular pressure data from rat nephrons in order to estimate the degree to which the myogenic dynamics of the afferent arteriole is modulated by the slower tubulo-glomerular dynamics. Our analysis reveals a significantly stronger interaction between the two mechanisms in spontaneously hypertensive rats than in normotensive rats.

Algorithms↗
PubMed2002

Does tissue oxygen-tension reliably reflect cerebral oxygen delivery and consumption?

We investigated the value of brain oxygen partial pressure (P(br)O(2)) with respect to predicting cerebral energetic failure in a rabbit model of global cerebral ischemia and hypoxia. Local cortical blood flow (l(co)CBF), P(br)O(2), extracellular lactate, pyruvate, and glutamate concentrations, as well as microvascular hemoglobin saturation (S(mv)O(2)), cytochrome oxidase redox level (Cyt a+a(3) oxidation), and brain electrical activity, were assessed during variable degrees of cerebral ischemia and hypoxia, induced by cisternal infusion of artificial cerebrospinal fluid or an admixture of nitrous oxide to inspiratory gas in 10 animals each. Arteriovenous difference in oxygen content, cerebral metabolic rate for oxygen, and oxygen extraction were derived from multimodal data. P(br)O(2), S(mv)O(2), and Cyt a+a(3) oxidation were closely related to cerebral blood flow and indices of oxidative metabolism. P(br)O(2) </=8 mm Hg corresponded to l(co)CBF </=15 mL. 100 g(-1). min(-1), S(mv)O(2) </=9%, Cyt a+a(3) oxidation </=20%, and progressive loss of brain electrical activity. Adequate tissue oxygenation was reflected by cerebral metabolic rate for oxygen >/=2.8 mL. 100 g(-1). min(-1), arteriovenous difference in oxygen content </=12.5 mL O(2). 100 mL(-1), and oxygen extraction </=60%. Meaningful interpretation of low P(br)O(2), especially with respect to definition of energetic thresholds, requires complementary information from simultaneous assessment of l(co)CBF and tissue oxygen extraction. IMPLICATIONS. The relationship between brain oxygen partial pressure and several variables of energy metabolism was investigated during variable degrees of cerebral ischemia and hypoxia in a rabbit model. Correct interpretation of individual brain oxygen partial pressure values, especially with respect to definition of energetic thresholds, requires complementary information from assessment of cerebral blood flow and tissue oxygen extraction.

Animals↗
PubMed2002

Functional brain imaging applications to differential diagnosis in the dementias.

PURPOSE OF REVIEW: The diagnosis of dementia rests on an improved knowledge and a better detection of early impairments, to which functional imaging can certainly contribute. RECENT FINDINGS: Progress has been observed at different levels. First, the understanding of different dementias has benefited from explorations of the neural substrate of dementia symptoms and from research into new markers. Second, diverse variables (clinical, anatomical, biochemical) have been related to impaired cerebral activity in Alzheimer's disease and other dementias, and progress in image analysis and in multimodal data acquisition has allowed a better understanding of the significance of brain activity disturbances. Third, functional imaging has been applied in well-designed clinical studies, and has provided important arguments for the diagnosis of characteristic clinical syndromes in the dementias. SUMMARY: The functioning of neural networks responsible for clinical symptoms in dementia remains an important research topic for functional imaging. The development of new tracers and new techniques for image processing should also improve the usefulness of brain imaging as a diagnostic tool.

Dementia↗
PubMed2026

Artificial intelligence in kidney cancer: a review of clinical applications across the disease spectrum.

PURPOSE OF REVIEW: This review examines recent advances (2024-2025) in the application of artificial intelligence (AI) to kidney cancer diagnosis, prognosis, and treatment planning. It categorizes studies across 13 clinical scenarios to assess where AI offers the most clinical utility. RECENT FINDINGS: AI models have demonstrated strong performance in a range of tasks including tumor grading, subtype classification, survival prediction, and risk stratification. Integration of radiomics, genomics, and histopathology has enabled personalized, noninvasive, and timely decision-making. The highest-performing models used CT-based radiomics, particularly for predicting progression-free and recurrence-free survival. However, performance varies across tasks and tumor subtypes, with lower accuracy in detecting oncocytomas or benign vs. malignant differentiation. AI applications in metastatic and nonresected cases remain underexplored, and ultrasound remains a largely under researched modality. While some models improve diagnostic accuracy and workflow efficiency, broader validation across diverse populations is still needed. SUMMARY: AI is transforming kidney cancer care across multiple clinical stages. Although promising, real-world implementation demands ongoing validation and postdeployment monitoring to prevent performance degradation due to distributional drift. AI's integration with multimodal data offers substantial potential to improve outcomes and reduce overtreatment.

Humans↗
PubMed2025

EWS::WT1 Isoform-Dependent Regulation of Neogenes in Desmoplastic Small Round Cell Tumors.

Desmoplastic small round cell tumor (DSRCT) is a rare, aggressive sarcoma characterized by the pathognomonic EWS::WT1 fusion protein (FP), an oncogenic chimeric transcription factor (OCTF) resulting from the t(11;22)(p13;q12) translocation. Recent studies have identified "neogenes" (NGs), genes normally silent in normal tissues but transcriptionally activated by OCTFs, as potential tumor-specific markers in fusion-driven cancers. In this study, we investigated the expression and regulation of DSRCT-specific NGs (DSRCT_NGs) using multimodal data across different cohorts of patients, PDX, and cell line data. We evaluated bulk and single-nucleus RNA sequencing of patient specimens from MD Anderson Cancer Center, revealing the robust ability for DSRCT_NGs to distinguish FP-positive DSRCT from samples failing detection of the EWS::WT1 FP. To elucidate the regulatory role of the EWS::WT1 FP in driving NG expression, we performed knockdown experiments in four DSRCT cell lines. This consistently resulted in a reduction of DSRCT_NG expression. Isoform-specific expression of EWS::WT1 in LP9 and MeT-5A mesothelial cells revealed that the E-KTS isoform of EWS::WT1 predominantly drives DSRCT_NG expression. Mechanistically, ATAC-seq and ChIP-seq analyses demonstrated that EWS::WT1 directly binds to accessible chromatin regions near NG transcription start sites, enriched for WT1 motifs and active histone marks. Integration of Hi-ChIP data further revealed that EWS::WT1 facilitates long-range enhancer-promoter looping at DSRCT_NG loci, promoting the expression of nearby genes. Collectively, these findings establish DSRCT_NGs as direct transcriptional outputs of the EWS::WT1 FP and implicate their loci as regulatory regions of the DSRCT transcriptome. Their fusion-dependent expression, chromatin accessibility, and promoter-enhancer connectivity underscore their potential utility as highly specific biomarkers and therapeutic targets in DSRCT.

DSRCT↗
PubMedDate not supplied

Open magnetic and electric graphic analysis.

The OMEGA software provides an analysis platform for user-independent, fast, and reproducible multimodal data analysis in one single software environment. Synergetic interactions pursued between the two functional imaging techniques fMRI and MEG use the morphological MRI recording as a basis for a common coordinate frame. In this way, direct interchange, comparison, and integration among the results of the different modalities have become feasible. The fMRI data analysis provides information about the localization of functional activity with low temporal resolution, whereas the MEG recording complements the corresponding time evolution with a high temporal resolution. The implementation of OMEGA allows the analyst to receive comprehensive MEG/fMRI results in a matter of minutes after the measurements have been completed. With OMEGA, the clinical researcher gets comprehensive information in a quick and standardized approach about the sites and the time course of neurological activation, which is useful for clinical applications and diagnostics.

Algorithms↗
PubMed2001

Registration of magnetic resonance spectroscopic imaging to computed tomography for radiotherapy treatment planning.

The incorporation of multiple imaging modalities into radiotherapy treatment planning offers the potential to improve identification of regions of pathology. This work outlines and evaluates a methodology for registration of magnetic resonance images (MRI) and spectroscopic images (MRSI) to computed tomography (CT) images, and visualization of the multimodality data on the treatment planning workstation. Volumetric magnetic resonance images were acquired during an examination prior to the initiation of radiotherapy. Registration between these images and the treatment planning computed tomography images was performed using an automated alignment routine, and was improved manually using an interactive registration tool. The parameters of the alignment were then used to transform the spectroscopic images into the same reference frame. The spectroscopy data were represented in terms of a statistical measure of abnormality, and embedded within the MRI data as overlaid contours. These images were sent via DICOM transfer to the treatment planning workstation. An analysis of the reproducibility of the

Brain Neoplasms↗
PubMed2005

Cerebral perfusion pressure and risk of brain hypoxia in severe head injury: a prospective observational study.

INTRODUCTION: Higher and lower cerebral perfusion pressure (CPP) thresholds have been proposed to improve brain tissue oxygen pressure (PtiO2) and outcome. We study the distribution of hypoxic PtiO2 samples at different CPP thresholds, using prospective multimodality monitoring in patients with severe traumatic brain injury. METHODS: This is a prospective observational study of 22 severely head injured patients admitted to a neurosurgical critical care unit from whom multimodality data was collected during standard management directed at improving intracranial pressure, CPP and PtiO2. Local PtiO2 was continuously measured in uninjured areas and snapshot samples were collected hourly and analyzed in relation to simultaneous CPP. Other variables that influence tissue oxygen availability, mainly arterial oxygen saturation, end tidal carbon dioxide, body temperature and effective hemoglobin, were also monitored to keep them stable in order to avoid non-ischemic hypoxia. RESULTS: Our main results indicate that half of PtiO2 samples were at risk of hypoxia (defined by a PtiO2 equal to or less than 15 mmHg) when CPP was below 60 mmHg, and that this percentage decreased to 25% and 10% when CPP was between 60 and 70 mmHg and above 70 mmHg, respectively (p < 0.01). CONCLUSION: Our study indicates that the risk of brain tissue hypoxia in severely head injured patients could be really high when CPP is below the normally recommended threshold of 60 mmHg, is still elevated when CPP is slightly over it, but decreases at CPP values above it.

Adult↗
PubMed2000

An overview of computer-integrated surgery and therapy.

Computer-integrated surgery and therapy (CIST): Methods and systems to help the surgeon or the physician use multimodality data (mainly medical images) in a rational and quantitative way, in order to plan but also to perform medical interventions through the use of passive, semi-active, or active guiding systems.

Diagnostic Imaging↗
PubMed1998

[The method and development of computer-assisted surgery].

The methodology and the state of the art of Computer-Assisted Surgery (CAS) are introduced in this paper. Computer-assisted surgery is a new high technology which uses computer science, biomedical engineering, mechanism, mathematics, surgery, and so on. Its objective is to help surgeons use multimodal data, such as CT, MRI, DSA, PET, et al. in a rational and quantitative way in order to plan and perform medical intervention. Stereotactic localization method and registration are two cruxes in computer-assisted surgery. There are several methods for localization and registration. In recent ten years, computer-assisted surgery has been a cynosure of scientists. Some computer-assisted surgery systems have been used in clinical practice.

General Surgery↗