Search PubMed⌕ Search

SEARCH · Search PubMed

Results for “Cross-modal alignment”

Search indexed PubMed citations on genomics, clinical trials, systematic reviews and public health. Explore titles, authors and supplied subject terms, then open the PubMed record.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 19 recordsLinked to original sources
PubMed2026

A multi-modal survival prediction framework with group-based batch training and structural consistency alignment.

OBJECTIVE: Integrating whole-slide images (WSIs) with transcriptomic profiles is pivotal for enhancing cancer survival prediction. However, the intrinsic gigapixel resolution and variable sequence lengths of WSIs create a fundamental trade-off between training efficiency and the preservation of data heterogeneity in existing frameworks. Furthermore, substantial statistical and structural discrepancies between histological and genomic modalities often impede effective cross-modal alignment and fusion, thereby limiting prognostic accuracy. METHODS: We propose PRISM, an efficient multi-modal learning framework for integrating WSIs with transcriptomic profiles. To reconcile training efficiency with full data heterogeneity, PRISM first stochastically partitions variable-length WSI sequences into a main subset and a complementary residual subset, both of which are packed into fixed-length groups for batch training. The main subset is processed in the main branch, utilizing isolation masking to maintain intra-group sequence independence. Simultaneously, the residual subset is consolidated into "hyperslides" within a residual branch that leverages tailored supervision, effectively capturing inter-slide correlations. Furthermore, PRISM integrates an Informative Token Aggregation (ITA) module to reduce redundancy in WSIs and employs Cross-batch Structural Consistency Alignment (CBSCA) mechanism to enhance inter-modal structural connectivity. Finally, efficient cross-modal feature interaction is achieved through a Low-rank Bilinear Gated Fusion (LBGF) module. Code is available at https://github.com/Alisa2080/PRISM. RESULTS: Compared with existing methods, PRISM achieves the best overall C-index across five TCGA cohorts. On the larger TCGA-BRCA dataset, PRISM requires only 6 hours of training time, substantially reducing computational cost relative to strong multimodal baselines. Furthermore, comprehensive evaluations demonstrate that PRISM achieves the best overall IBS ranking and favorable time-dependent AUC performance at 1, 3, and 5 years, thereby delivering a more favorable trade-off between prognostic performance and computational efficiency. CONCLUSION: PRISM provides a favorable balance between predictive performance, calibration quality, and computational efficiency, highlighting its potential for practical deployment in multimodal survival modeling for computational pathology.

Humans↗
PubMed2026

Unifying multimodal single-cell data with a mixture-of-experts β-variational autoencoder framework.

Multimodal single-cell assays profile complementary layers of cell state, but integration is complicated by modality mismatch, sparsity, and uneven cohort coverage. Here, we present Unified Variational Inference (UniVI), a scalable mixture-of-experts β-variational autoencoder that learns a shared latent space while preserving modality-specific structure. UniVI couples modality-specific encoders/decoders with a shared latent prior and a symmetric cross-modal alignment objective, enabling consistent integration of paired measurements without curated feature-link graphs or preannotated reference atlases; optional supervised heads can be added when labels are available. Across paired RNA-protein (CITE-seq) and RNA-chromatin (10x Genomics Multiome, SHARE-seq) data spanning human PBMCs and mouse back skin-a nonhematopoietic tissue with continuous differentiation hierarchies-UniVI produces coherent embeddings, improves label transfer, and enables cross-modal reconstruction and denoising. Extending to trimodal measurements, UniVI maintains robust three-way alignment among RNA, chromatin accessibility, and surface proteins (TEA-seq), and accommodates DNA methylation in a paired scNMT-seq mouse gastrulation proof-of-concept under beta-binomial likelihoods. Performance degrades gracefully under severe cell type imbalance and in the presence of modality-exclusive populations. In an acute myeloid leukemia mosaic design, a paired RNA-protein bridge anchors independent RNA-only and protein+genotype cohorts, revealing genotype-associated neighborhoods that sharpen with mutation-aware fine-tuning. UniVI thus provides a flexible, interpretable framework for multimodal integration across paired, trimodal, and mosaic study designs and supports practical reference-to-query projection in partially observed studies.

Journal Article↗
PubMed2026

Integrating histology and spatial transcriptomics via multimodal transformers and contrastive representation learning for accurate gene expression prediction.

Predicting spatial gene expression from Histological images is a fundamental task in understanding tissue organization and molecular phenotypes. However, existing methods often rely on single-model representations or lack effective alignment between image and transcriptomic features. To address these limitations, we propose a unified multimodal learning framework that integrates histological imaging and spatial transcriptomics through a shared latent representation space. Specifically, histological H&E images are encoded by a ResNet50-based convolutional stem and a MobileViT Transformer backbone to extract hierarchical visual representations. Both modalities are projected into a shared latent space via linear-GELU-dropout transformation blocks, enabling cross-modal alignment through a contrastive learning objective that maximizes agreement between the corresponding image and the spot embeddings. Experimental results on the 10x Genomics Visium dataset of human liver tissue demonstrate that MViTGene achieves significantly higher prediction accuracy than existing methods across multiple gene subsets, with improvements of 20%, 33%, and 12% in predicting marker genes, highly expressed genes, and highly variable genes, respectively. The significant improvement in relevance indicates that the model can more accurately capture the true correspondence between tissue morphology and gene expression, therefore enabling more reliable biological interpretation. It provides a computational tool for high-throughput spatial gene expression prediction that balances performance and interpretability.

Humans↗
PubMed2026

SIVA: diagonal integration of spatial multi-omics data via spatially informed variational autoencoders and anchor guidance.

MOTIVATION: Understanding cellular states and regulatory programs requires integrative analysis of multiple omics layers. Although recent spatial sequencing technologies allow molecular profiling of cells within their tissue context, paired spatial multi-omics assays are still limited by technical complexity and cost. This creates a pressing need for diagonal integration methods that enable joint analysis of unpaired spatial omics datasets. RESULTS: We propose SIVA, a deep generative framework based on Spatially-Informed Variational Autoencoders with Anchor Guidance, for diagonal integration of spatial multi-modal data. SIVA employs modality-specific variational autoencoders (VAEs) with a hybrid latent embedding that integrates Gaussian process and standard Gaussian priors, enabling joint modeling of spatially structured variation and dominant underlying data distributions across modalities. To facilitate cross-modal alignment in the absence of one-to-one cell correspondence, SIVA adopts a dual integration strategy combining global distribution alignment via Maximum Mean Discrepancy and local correspondence guidance using mutual nearest neighbor anchors. Extensive experiments across multiple cross-slice integration scenarios demonstrate that SIVA achieves robust and accurate integration of unpaired spatial omics datasets, consistently outperforming existing methods. AVAILABILITY AND IMPLEMENTATION: The source codes are available at https://github.com/PelenJiang/SIVA.

Autoencoder↗
PubMed2026

ARISE: RNA-anchored shared-edge topology and hierarchical fusion for spatial multi-omics integration.

MOTIVATION: Spatial multi-omics technologies jointly profile transcriptomes, proteins and chromatin accessibility in situ, enabling integrative analysis of tissue organization across molecular layers. However, most existing graph-based integration methods rely on independently constructed modality-specific k-nearest-neighbor graphs. When auxiliary modalities are sparse or noisy, these graphs can become topologically discordant, propagate spurious edges, weaken cross-modal alignment, and reduce spatial domain resolution. RESULTS: We present Anchored RNA for Integrated Spatial Embedding (ARISE), an RNA expression anchored framework for spatial multi-omics integration. ARISE defines a shared-edge topology by intersecting RNA feature-similarity and spatial-proximity graphs, encodes auxiliary modalities on this common scaffold, and integrates them through inside-out hierarchical fusion. We further show theoretically that graph intersection minimizes false-positive edges within a broad class of k-of-r graph fusion rules, providing a principled basis for topology anchoring. Across various spatial multi-omics benchmarks spanning simulated and real datasets in bi-modal and tri-modal settings, ARISE improves spatial domain identification, cross-modal consistency, and preservation of tissue structure relative to existing methods. Furthermore, the learned representation supports biologically meaningful downstream analyses, including marker-based domain annotation, pathway enrichment, and cis-regulatory inference, indicating that ARISE yields a robust and interpretable framework for spatial multi-omics integration. AVAILABILITY AND IMPLEMENTATION: The source code is available at https://github.com/XiangxiangWang-code/ARISE. The archived version used in this study is available at https://doi.org/10.6084/m9.figshare.32686137.v2.

Multiomics↗
PubMed2000

Left tactile extinction following visual stimulation of a rubber hand.

In close analogy with neurophysiological findings in monkeys, neuropsychological studies have shown that the human brain constructs visual maps of space surrounding different body parts. In right-brain-damaged patients with tactile extinction, the existence of a visual peripersonal space centred on the hand has been demonstrated by showing that cross-modal visual-tactile extinction is segregated mainly in the space near the hand. That is, tactile stimuli on the contralesional hand are extinguished more consistently by visual stimuli presented near the ipsilesional hand than those presented far from it. Here, we report the first evidence in humans that this hand-centred visual peripersonal space can be coded in relation to a seen rubber replica of the hand, as if it were a real hand. In patients with left tactile extinction, a visual stimulus presented near a seen right rubber hand induced strong cross-modal visual-tactile extinction, similar to that obtained by presenting the same visual stimulus near the patient's right hand. Critically, this specific cross-modal effect was evident when subjects saw the rubber hand as having a plausible posture relative to their own body (i.e. when it was aligned with the subject's right shoulder). In contrast, cross-modal extinction was strongly reduced when the seen rubber hand was arranged in an implausible posture (i. e. misaligned with respect to the subject's right shoulder). We suggest that this phenomenon is due to the dominance of vision over proprioception: the system coding peripersonal space can be 'deceived' by the vision of a fake hand, provided that its appearance looks plausible with respect to the subject's body.

Aged↗
PubMed2003

Visual localization ability influences cross-modal bias.

The ability of a visual signal to influence the localization of an auditory target (i.e., "cross-modal bias") was examined as a function of the spatial disparity between the two stimuli and their absolute locations in space. Three experimental issues were examined: (a) the effect of a spatially disparate visual stimulus on auditory localization judgments; (b) how the ability to localize visual, auditory, and spatially aligned multisensory (visual-auditory) targets is related to cross-modal bias, and (c) the relationship between the magnitude of cross-modal bias and the perception that the two stimuli are spatially "unified" (i.e., originate from the same location). Whereas variability in localization of auditory targets was large and fairly uniform for all tested locations, variability in localizing visual or spatially aligned multisensory targets was much smaller, and increased with increasing distance from the midline. This trend proved to be strongly correlated with biasing effectiveness, for although visual-auditory bias was unexpectedly large in all conditions tested, it decreased progressively (as localization variability increased) with increasing distance from the midline. Thus, central visual stimuli had a substantially greater biasing effect on auditory target localization than did more peripheral visual stimuli. It was also apparent that cross-modal bias decreased as the degree of visual-auditory disparity increased. Consequently, the greatest visual-auditory biases were obtained with small disparities at central locations. In all cases, the magnitude of these biases covaried with judgments of spatial unity. The results suggest that functional properties of the visual system play the predominant role in determining these visual-auditory interactions and that cross-modal biases can be substantially greater than previously noted.

Adult↗
PubMed2026

ARCADIA reveals spatially dependent transcriptional programs through integration of scRNA-seq and spatial proteomics.

MOTIVATION: Cellular states are strongly influenced by spatial context, but single-cell RNA sequencing (scRNA-seq) loses information about local tissue organization, while spatial proteomic assays capture limited marker panels that constrain transcriptomic inference. Integrating these modalities can elucidate how spatial niches shape transcriptional programs, yet existing approaches depend on either feature-level correspondence such as gene-protein linkage or cell-level barcode pairing, which is often unavailable. RESULTS: We present ARCADIA (ARchetype-based Clustering and Alignment with Dual Integrative Autoencoders), a generative framework for cross-modal integration that operates without cell barcode pairing and does not assume direct feature-to-feature correspondence. ARCADIA identifies modality-specific archetypes, that is, convex combinations of cells representing extreme phenotypic states, and aligns these anchors across modalities by minimizing the discrepancy between their cell-type composition profiles. The aligned archetypes define a shared coordinate system that anchors dual variational autoencoders (VAEs) trained with cross-modal geometric regularization, preserving archetype structure and spatial neighborhood information while enabling bidirectional translation between modalities. On semi-synthetic CITE-seq data, ARCADIA outperforms existing weak-linkage methods. Applied to independent human tonsil scRNA-seq and CODEX data, ARCADIA reconstructs known tissue architecture and reveals spatially dependent transcriptional programs linking B-cell maturation and T-cell activation or exhaustion to microenvironmental niches. AVAILABILITY AND IMPLEMENTATION: Source code is accessible at https://github.com/azizilab/ARCADIA_public. Reproducibility scripts and data are available at https://github.com/azizilab/arcadia_reproducibility.

Proteomics↗
PubMedDate not supplied

MRI-PET registration with automated algorithm.

OBJECTIVE: We have previously reported an automated method for within-modality (e.g., PET-to-PET) image alignment. We now describe modifications to this method that allow for cross-modality registration of MRI and PET brain images obtained from a single subject. METHODS: This method does not require fiducial markers and the user is not required to identify common structures on the two image sets. To align the images, the algorithm seeks to minimize the standard deviation of the PET pixel values that correspond to each MRI pixel value. The MR images must be edited to exclude nonbrain regions prior to using the algorithm. RESULTS AND CONCLUSION: The method has been validated quantitatively using data from patients with stereotaxic fiducial markers rigidly fixed in the skull. Maximal three-dimensional errors of < 3 mm and mean three-dimensional errors of < 2 mm were measured. Computation time on a SPARCstation IPX varies from 3 to 9 min to align MR image sets with [18F]fluorodeoxyglucose PET images. The MR alignment with noisy H2(15)O PET images typically requires 20-30 min.

Adult↗
PubMed2026

NeuroOmics-Net: An interpretable multimodal deep learning framework for Alzheimer's disease diagnosis and progression prediction using neuroimaging, EEG, and genomic data.

Accurate diagnosis and progression prediction of Alzheimer's disease (AD) remain challenging due to the heterogeneous nature of the disease, which involves structural brain degeneration, electrophysiological dysfunction, and molecular dysregulation. Most existing deep learning approaches rely on a single modality or limited multimodal combinations, thereby failing to capture the complex cross-domain interactions underlying AD progression. Furthermore, the scarcity of large-scale datasets containing synchronized neuroimaging, electrophysiological, and genomic measurements restricts the development of comprehensive multimodal diagnostic systems. To address these challenges, this study proposes NeuroOmics-Net, a multimodal deep learning framework for Alzheimer's disease analysis that integrates structural magnetic resonance imaging (sMRI), electroencephalography (EEG), and gene expression data. The proposed framework combines a Hierarchical Multi-View Encoder (HME) for modality-specific feature extraction, a Cross-Omics Attention Fusion (CAF) module for adaptive integration of complementary biomarkers, and a Disease Progression Graph Learning (DPGL) module for modeling progression-related relationships across biological domains. To facilitate cross-modal integration from independent cohorts, Regularized Canonical Correlation Analysis (RCCA) is employed to align heterogeneous feature representations within a shared latent space. Experiments were conducted using publicly available datasets from ADNI, PhysioNet, and GEO repositories comprising 1120 diagnosis-aligned samples. The proposed framework achieved 94.3% classification accuracy and an AUC of 0.975 for distinguishing normal controls (NC), mild cognitive impairment (MCI), and Alzheimer's disease subjects, while attaining 93.7% accuracy for predicting conversion from stable mild cognitive impairment (sMCI) to progressive mild cognitive impairment (pMCI). However, a fairness sensitivity analysis using stratified demographic reweighting revealed accuracy ranging from 90.8% (low-education, high-comorbidity proxy subgroup) to 96.1% (low-risk, high-reserve proxy subgroup), a demographic parity gap of 5.3 percentage points, indicating that overall accuracy reflects a performance ceiling in a relatively homogeneous research cohort rather than a realistic estimate for demographically diverse clinical populations. Comparative evaluations demonstrated consistent improvements over state-of-the-art unimodal and multimodal deep learning models. Interpretability analysis further identified clinically relevant biomarkers, including hippocampal and entorhinal atrophy, theta-alpha EEG alterations, and APOE-associated molecular pathways. Because sMRI, EEG, and gene expression data were sourced from separate, unpaired cohorts with no subjects possessing all three synchronized measurements, all reported cross-modal associations reflect population-level statistical correspondence across diagnosis-matched groups rather than within-subject physiological coupling; no claim of intra-individual causal cross-modal interaction is made. These findings demonstrate that NeuroOmics-Net provides an effective computer-aided framework for multimodal biomedical data processing and Alzheimer's disease analysis. By integrating neuroimaging, electrophysiological, and genomic information, the proposed approach enables accurate diagnosis, progression prediction, and biologically interpretable decision support for clinical and translational applications.

Humans↗
PubMed2003

The "Flash-Lag" effect occurs in audition and cross-modally.

In 1958 MacKay showed that a rigidly moving object becomes visually fragmented when part of it is continuously visible but the rest is illuminated intermittently. For example, the glowing tip of a lit cigarette moving under stroboscopic illumination appeared to move ahead of the intermittently lit body. Latterly rediscovered as "the flash-lag effect" (FLE), this illusion now is typically demonstrated on a computer monitor showing two spots of light, one translating across the screen and another briefly flashed in vertical alignment with it. Despite being physically aligned, the brief flash is seen to lag behind the moving spot. This effect has recently motivated much fruitful research, prompting a variety of potential explanations, including those based on motion extrapolation, differential latency, attention, postdiction, and temporal integration (for review, see ). With no consensus on which theory is most plausible, we have broadened the scope of enquiry to include audition and have found that the FLE is not confined to vision. Whether the auditory motion stimulus is a frequency sweep or a translating sound source, briefly presented auditory stimuli lag behind auditory movement. In addition, when we used spatial motion, we found that the FLE can occur cross-modally. Together, these findings challenge several FLE theories and point to a discrepancy between internal brain timing and external stimulus timing.

Auditory Perception↗
PubMed2000

Fusion of coregistered cross-modality images using a temporally alternating display method.

As an aid to the interpretation of functional images, cross-modality coregistration of functional and anatomical images has grown rapidly. Various ways of easily interpreting and visualising coregistered images have previously been investigated; for their display, an intensity-weighted temporally alternating method is used. For brain images, geometric registration involves the automatic alignment method, using the head scalp boundary extracted from the sinogram of a PET emission scan and a surface-matching algorithm; images of the chest or abdomen are registered semi-automatically using a paired point matching algorithm. For the simultaneous display of geometrically registered images, rapid image switching is applied; both images are written with independent colour scales. The rapidly alternating display of two images, synchronised with monitor scanning, induces the fusion of images in the human visual perception system. The accuracy of registration of PET and MRI images is within 2 mm for two point sets. A resulting image is intensified by weighting the display time and/or controlling the intensity map of each image with the degree of interest. This method may be useful for the interpretation and visualisation of coregistered images.

Algorithms↗
PubMed2003

Cross-modal perceptual integration of spatially and temporally disparate auditory and visual stimuli.

Under certain conditions, auditory and visual information are integrated into a single unified percept even when they originate in different locations in space. The present study shows how this illusion, known as the ventriloquism effect, depends on spatial, temporal and cognitive factors. A method of psychophysical scaling was employed in combination with simple auditory-visual stimuli (tone bursts and flashing light spots) that were presented with various spatiotemporal disparities. Participants either judged their impression of the likelihood of a common cause (Experiment 1) or spatial alignment (Experiment 2) or synchrony of sound and light (Experiment 3). In all three experiments the participants' judgements depended significantly on temporal disparity whereas influences of spatial disparity were significant in Experiments 1 and 2. Optimum scores were always obtained when auditory stimuli were presented with a delay of 50-100 ms after the visual stimuli. These results demonstrate that both temporal and spatial proximity of the two stimuli are critical for the experience of phenomenal causality. On the other hand, spatio-temporal ranges for optimal perception of phenomenal causality in Experiment 1 were significantly larger than predicted by simultaneous detection of spatial and temporal disparities. This finding suggests that auditory-visual binding was further facilitated by additional, cognitive, factors, associated with the specific instruction to judge the likelihood of a common cause. Obviously, these instructional influences may reflect similar perceptual effects, as have been shown previously by increasing the complexity or cognitive compellingness of auditory-visual stimuli.

Acoustic Stimulation↗
PubMed1985

Confusion of laterally inverted mirror-images: a relationship to brain anatomy?

Whereas in the monkey brain the representation of spatial direction (left/ right) is the same for visual and for tactual inflow, in the human brain visual and tactual lateral directions are not aligned with respect to one another. This anatomical feature of the human brain may account in part for the particular difficulty young sighted children have with laterally inverted forms. A small group of children who were totally blind since birth distinguished mirror pairs by touch significantly more easily than did sighted children of comparable age; and monkeys succeeded at cross-modal recognition of laterally inverted mirror pairs better (relative to control pairs) than did sighted children.

Age Factors↗
PubMed1997

Display of coregistered cross-modality images using time-weighted alternation method.

Cross-modality coregistration of functional and anatomical images has rapidly grown as the aid for the interpretation of functional images. Various methods have been previously investigated so that the coregistered images can be easily interpreted and realistically visualized. We used the time/intensity-weighted alternation method to display the coregistered images simultaneously. Stereotactic registration was done with two methods. We realized the automated method of alignment using the boundary of head scalp extracted from the sinogram of a PET emission scan and surface-matching algorithm in brain, and the semi-automated method using a paired point-matching algorithm in the chest and abdomen. For the simultaneous display of the stereotactically registered images, rapid switching of the screens were applied on which both images were written with independent color scales. Rapid alternate display of two images, that is synchronized using the monitor scanning process, could induce the fusion of images in the human visual perception system. Also, intensification of an interesting image is possible by weighting the displaying time or global intensities of images with the degree of interest. These techniques could be useful for the interpretation and visualization of the coregistered images.

Brain↗
PubMed1996

Representation and integration of multiple sensory inputs in primate superior colliculus.

1. The properties of visual-, auditory-, and somatosensory-responsive neurons, as well as of neurons responsive to multiple sensory cues (i.e., multisensory), were examined in the superior colliculus of the rhesus monkey. Although superficial layer neurons responded exclusively to visual stimuli and visual inputs predominated in deeper layers, there was also a rich nonvisual and multisensory representation in the superior colliculus. More than a quarter (27.8%) of the deep layer population responded to stimuli from more than a single sensory modality. In contrast, 37% responded only to visual cues, 17.6% to auditory cues, and 17.6% to somatosensory cues. Unimodal- and multisensory-responsive neurons were clustered by modality. Each of these modalities was represented in map-like fashion, and the different representations were in alignment with one another. 2. Most deep layer visually responsive neurons were binocular and exhibited poor selectivity for such stimulus characteristics as orientation, velocity, and direction of movement. Similarly, most auditory-responsive neurons had contralateral receptive fields and were binaural, but had little frequency selectivity and preferred complex, broad-band sounds. Somatosensory-responsive neurons were overwhelmingly contralateral, high velocity, and rapidly adapting. Only rarely did somatosensory-responsive neurons require distortion of subcutaneous tissue for activation. 3. The spatial congruence among the different receptive fields of multisensory neurons was a critical feature underlying their ability to synthesize cross-modal information. 4. Combinations of stimuli could have very different consequences in the same neuron, depending on their temporal and spatial relationships. Generally, multisensory interactions were evident when pairs of stimuli were separated from one another by < 500 ms, and the products of these interactions far exceeded the sum of their unimodal components. Whether the combination of stimuli produced response enhancement, response depression, or no interaction depended on the location of the stimuli relative to one another and to their respective receptive fields. Maximal response enhancements were observed when stimuli originated from similar locations in space (as when derived from the same event) because they fell within the excitatory receptive fields of the same multisensory neurons. If, however, the stimuli were spatially disparate such that one fell beyond the excitatory borders of its receptive field, either no interaction was produced or this stimulus depressed the effectiveness of the other. Furthermore, maximal response interactions were seen with the pairing of weakly effective unimodal stimuli. As the individual unimodal stimuli became increasingly effective, the levels of response enhancement to stimulus combinations declined, a principle referred to as inverse effectiveness. Many of the integrative principles seen here in the primate superior colliculus are strikingly similar to those observed in the cat. These observations indicate that a set of common principles of multisensory integration is adaptable in widely divergent species living in very different ecological situations. 5. Surprisingly, a few multisensory neurons had individual receptive fields that were not in register with one another. This has not been noted in multisensory neurons of other species, and these "anomalous" receptive fields could present a daunting problem: stimuli originating from the same general location in space cannot simultaneously fall within their respective receptive fields, a stimulus pairing that may result in response depression. Conversely, stimuli that originate from separate events and disparate locations (and fall within their receptive fields) may result in response enhancement. However, the spatial principle of multisensory integration did not apply in these cases. (ABSTRACT TRUNCATED)

Animals↗
PubMed2000

Registration of planar emission images with reprojected CT data.

UNLABELLED: Planar gamma-camera imaging is still widely used clinically. Alignment of planar images with images from tomographic modalities, such as CT, or with other planar images would be desirable. Here, we present and evaluate a method for such an alignment, using planar transmission images acquired with the emission images and reprojection of the 3-dimensional CT data. This method permits determination of which CT slice corresponds to a particular row of pixels in the gamma-camera image and which column of pixels in that CT slice corresponds to a particular pixel in the emission data. METHODS: A method based on maximization of the correlation coefficient, previously used for 3-dimensional datasets, was modified to permit 2-dimensional registrations. Planar transmission measurements were obtained using a collimated 99mTc flood source in conjunction with planar emission studies. The CT data were first reprojected to permit the 2-dimensional registration. The registration method was evaluated for its accuracy and reproducibility. RESULTS: For phantom data, the registration errors were -0.1 +/- 1.0 mm for x-translations, 1.0 +/- 1.3 mm for y-translations, and -0.2 +/- 0.3 degrees for rotations. For patient data, the errors were 1.6 +/- 0.8 mm for x-translations, 1.3 +/- 1.0 mm for y-translations, and 0.5 +/- 0.5 degrees for rotations. An examination of the need for rescaling of the attenuation data (to compensate for the different photon energies used in the respective attenuation measurements) showed no significant impact on registration error. When 5 different regions of interest were used for the correlation coefficient calculation, the mean errors attributable to region-of-interest choice alone were 1.0 mm for x-translations, 2.0 mm for y-translations, and 1.2 degrees for rotations. CONCLUSION: In almost all instances, translational registration errors were kept to subpixel levels (pixel size, 2.6 mm) and rotational errors to 1 degrees or less. The 1 exception was in the easily avoidable case of "pitch" rotations of the patient of 2 degrees or more. The modified registration method provides a simple yet reliable way to provide cross-modality evaluation of planar emission data.

Algorithms↗
PubMedDate not supplied

A metric for testing the accuracy of cross-modality image registration: validation and application.

PURPOSE: Our goals were to (a) develop and validate a sensitive measure of image registration, applicable between as well as within imaging modalities; and (b) quantify the accuracy of the automated image registration (AIR) algorithm in retrospectively registering MR and PET images of baboon brain. METHOD: We studied five monkeys, each with a surgically implanted "cap" that was bolted to an acrylic headholder containing fiducial markers. Anatomic MRI and PET H2 15O blood flow images were aligned using AIR (ignoring the fiducials). RESULTS: (a) Fiducial points were localized to about one-tenth the voxel size. Distances computed from the fiducial points were correct to within 0.2% (MRI) and 1.4% (PET). (b) The mean error remaining after AIR, measured at the five fiducial points, varied from 2.9 to 5.3 mm. The average registration error within the brain was calculated to be 2.20 mm (maximum, 3.57 mm). This error was not primarily rotational about the center of the brain and was worsened by heavy smoothing of the PET images. CONCLUSION: (a) We have defined a reliable, sensitive metric that can stringently test the accuracy of various image registration techniques. (b) The AIR algorithm leaves modest error after aligning PET blood flow and anatomic MR images of baboon brain.

Algorithms↗