Search PubMed⌕ Search

SEARCH · Search PubMed

Results for “Cross-modal alignment”

Search indexed PubMed citations on genomics, clinical trials, systematic reviews and public health. Explore titles, authors and supplied subject terms, then open the PubMed record.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

25 records · Page 2Linked to original sources

Display of coregistered cross-modality images using time-weighted alternation method.

Cross-modality coregistration of functional and anatomical images has rapidly grown as the aid for the interpretation of functional images. Various methods have been previously investigated so that the coregistered images can be easily interpreted and realistically visualized. We used the time/intensity-weighted alternation method to display the coregistered images simultaneously. Stereotactic registration was done with two methods. We realized the automated method of alignment using the boundary of head scalp extracted from the sinogram of a PET emission scan and surface-matching algorithm in brain, and the semi-automated method using a paired point-matching algorithm in the chest and abdomen. For the simultaneous display of the stereotactically registered images, rapid switching of the screens were applied on which both images were written with independent color scales. Rapid alternate display of two images, that is synchronized using the monitor scanning process, could induce the fusion of images in the human visual perception system. Also, intensification of an interesting image is possible by weighting the displaying time or global intensities of images with the degree of interest. These techniques could be useful for the interpretation and visualization of the coregistered images.

Brain↗

Representation and integration of multiple sensory inputs in primate superior colliculus.

1. The properties of visual-, auditory-, and somatosensory-responsive neurons, as well as of neurons responsive to multiple sensory cues (i.e., multisensory), were examined in the superior colliculus of the rhesus monkey. Although superficial layer neurons responded exclusively to visual stimuli and visual inputs predominated in deeper layers, there was also a rich nonvisual and multisensory representation in the superior colliculus. More than a quarter (27.8%) of the deep layer population responded to stimuli from more than a single sensory modality. In contrast, 37% responded only to visual cues, 17.6% to auditory cues, and 17.6% to somatosensory cues. Unimodal- and multisensory-responsive neurons were clustered by modality. Each of these modalities was represented in map-like fashion, and the different representations were in alignment with one another. 2. Most deep layer visually responsive neurons were binocular and exhibited poor selectivity for such stimulus characteristics as orientation, velocity, and direction of movement. Similarly, most auditory-responsive neurons had contralateral receptive fields and were binaural, but had little frequency selectivity and preferred complex, broad-band sounds. Somatosensory-responsive neurons were overwhelmingly contralateral, high velocity, and rapidly adapting. Only rarely did somatosensory-responsive neurons require distortion of subcutaneous tissue for activation. 3. The spatial congruence among the different receptive fields of multisensory neurons was a critical feature underlying their ability to synthesize cross-modal information. 4. Combinations of stimuli could have very different consequences in the same neuron, depending on their temporal and spatial relationships. Generally, multisensory interactions were evident when pairs of stimuli were separated from one another by < 500 ms, and the products of these interactions far exceeded the sum of their unimodal components. Whether the combination of stimuli produced response enhancement, response depression, or no interaction depended on the location of the stimuli relative to one another and to their respective receptive fields. Maximal response enhancements were observed when stimuli originated from similar locations in space (as when derived from the same event) because they fell within the excitatory receptive fields of the same multisensory neurons. If, however, the stimuli were spatially disparate such that one fell beyond the excitatory borders of its receptive field, either no interaction was produced or this stimulus depressed the effectiveness of the other. Furthermore, maximal response interactions were seen with the pairing of weakly effective unimodal stimuli. As the individual unimodal stimuli became increasingly effective, the levels of response enhancement to stimulus combinations declined, a principle referred to as inverse effectiveness. Many of the integrative principles seen here in the primate superior colliculus are strikingly similar to those observed in the cat. These observations indicate that a set of common principles of multisensory integration is adaptable in widely divergent species living in very different ecological situations. 5. Surprisingly, a few multisensory neurons had individual receptive fields that were not in register with one another. This has not been noted in multisensory neurons of other species, and these "anomalous" receptive fields could present a daunting problem: stimuli originating from the same general location in space cannot simultaneously fall within their respective receptive fields, a stimulus pairing that may result in response depression. Conversely, stimuli that originate from separate events and disparate locations (and fall within their receptive fields) may result in response enhancement. However, the spatial principle of multisensory integration did not apply in these cases. (ABSTRACT TRUNCATED)

Animals↗

Registration of planar emission images with reprojected CT data.

UNLABELLED: Planar gamma-camera imaging is still widely used clinically. Alignment of planar images with images from tomographic modalities, such as CT, or with other planar images would be desirable. Here, we present and evaluate a method for such an alignment, using planar transmission images acquired with the emission images and reprojection of the 3-dimensional CT data. This method permits determination of which CT slice corresponds to a particular row of pixels in the gamma-camera image and which column of pixels in that CT slice corresponds to a particular pixel in the emission data. METHODS: A method based on maximization of the correlation coefficient, previously used for 3-dimensional datasets, was modified to permit 2-dimensional registrations. Planar transmission measurements were obtained using a collimated 99mTc flood source in conjunction with planar emission studies. The CT data were first reprojected to permit the 2-dimensional registration. The registration method was evaluated for its accuracy and reproducibility. RESULTS: For phantom data, the registration errors were -0.1 +/- 1.0 mm for x-translations, 1.0 +/- 1.3 mm for y-translations, and -0.2 +/- 0.3 degrees for rotations. For patient data, the errors were 1.6 +/- 0.8 mm for x-translations, 1.3 +/- 1.0 mm for y-translations, and 0.5 +/- 0.5 degrees for rotations. An examination of the need for rescaling of the attenuation data (to compensate for the different photon energies used in the respective attenuation measurements) showed no significant impact on registration error. When 5 different regions of interest were used for the correlation coefficient calculation, the mean errors attributable to region-of-interest choice alone were 1.0 mm for x-translations, 2.0 mm for y-translations, and 1.2 degrees for rotations. CONCLUSION: In almost all instances, translational registration errors were kept to subpixel levels (pixel size, 2.6 mm) and rotational errors to 1 degrees or less. The 1 exception was in the easily avoidable case of "pitch" rotations of the patient of 2 degrees or more. The modified registration method provides a simple yet reliable way to provide cross-modality evaluation of planar emission data.

Algorithms↗

Perceived timing of first- and second-order changes in vision and hearing.

Simultaneous changes in visual stimulus attributes (such as motion or color) are often perceived to occur at different times, a fact usually attributed to differences in neural processing times of those attributes. However, other studies suggest that perceptual misalignments are not due to stimulus attributes, but to the type of change, first- or second-order. To test whether this idea generalizes across modalities, we studied perceptual synchrony of acoustic and of audiovisual cross-modal stimuli, which varied in a first- or second-order fashion. First-order changes were abrupt changes in tone intensity or frequency (auditory), or spatial position (visual), while second-order changes were an inversion of the direction of change, such as a turning point when a rising tone starts falling or a translating visual blob reverses. For both pure acoustic and cross-modal stimuli, first-order changes were systematically perceived before second-order changes. However, when both changes were first-order, or both were second-order, little or no difference in perceptual delay was found between them, regardless of attribute or modality. This shows that the type of attribute change, as well as latency differences, is a strong determinant of subjective temporal alignments. We also performed an analysis of reaction times (RTs) to the first- and second-order attribute changes used in these temporal alignment experiments. RT differences between these stimuli did not correspond with our temporal alignment data, suggesting that subjective alignments cannot be accounted for by a simple latency-based explanation.

Acoustic Stimulation↗

Good times for multisensory integration: Effects of the precision of temporal synchrony as revealed by gamma-band oscillations.

The synchronous occurrence of the unisensory components of a multisensory stimulus contributes to their successful merging into a coherent perceptual representation. Oscillatory gamma-band responses (GBRs, 30-80 Hz) have been linked to feature integration mechanisms and to multisensory processing, suggesting they may also be sensitive to the temporal alignment of multisensory stimulus components. Here we examined the effects on early oscillatory GBR brain activity of varying the precision of the temporal synchrony of the unisensory components of an audio-visual stimulus. Audio-visual stimuli were presented with stimulus onset asynchronies ranging from -125 to +125 ms. Randomized streams of auditory (A), visual (V), and audio-visual (AV) stimuli were presented centrally while subjects attended to either the auditory or visual modality to detect occasional targets. GBRs to auditory and visual components of multisensory AV stimuli were extracted for five subranges of asynchrony (e.g., A preceded by V by 100+/-25 ms, by 50+/-25 ms, etc.) and compared with GBRs to unisensory control stimuli. Robust multisensory interactions were observed in the early GBRs when the auditory and visual stimuli were presented with the closest synchrony. These effects were found over medial-frontal brain areas after 30-80 ms and over occipital brain areas after 60-120 ms. A second integration effect, possibly reflecting the perceptual separation of the two sensory inputs, was found over occipital areas when auditory inputs preceded visual by 100+/-25 ms. No significant interactions were observed for the other subranges of asynchrony. These results show that the precision of temporal synchrony can have an impact on early cross-modal interactions in human cortex.

Acoustic Stimulation↗

A metric for testing the accuracy of cross-modality image registration: validation and application.

PURPOSE: Our goals were to (a) develop and validate a sensitive measure of image registration, applicable between as well as within imaging modalities; and (b) quantify the accuracy of the automated image registration (AIR) algorithm in retrospectively registering MR and PET images of baboon brain. METHOD: We studied five monkeys, each with a surgically implanted "cap" that was bolted to an acrylic headholder containing fiducial markers. Anatomic MRI and PET H2 15O blood flow images were aligned using AIR (ignoring the fiducials). RESULTS: (a) Fiducial points were localized to about one-tenth the voxel size. Distances computed from the fiducial points were correct to within 0.2% (MRI) and 1.4% (PET). (b) The mean error remaining after AIR, measured at the five fiducial points, varied from 2.9 to 5.3 mm. The average registration error within the brain was calculated to be 2.20 mm (maximum, 3.57 mm). This error was not primarily rotational about the center of the brain and was worsened by heavy smoothing of the PET images. CONCLUSION: (a) We have defined a reliable, sensitive metric that can stringently test the accuracy of various image registration techniques. (b) The AIR algorithm leaves modest error after aligning PET blood flow and anatomic MR images of baboon brain.

Algorithms↗

scMultiNODE: Integrative and Scalable Framework for Multi-Modal Temporal Single-Cell Data.

Measuring single-cell genomic profiles at different timepoints enables our understanding of cell development. This understanding is more comprehensive when we perform an integrative analysis of multiple measurements (or modalities) across various developmental stages. However, obtaining such measurements from the same set of single cells is resource-intensive, restricting our ability to study them jointly. We introduce scMultiNODE, an unsupervised integration model that combines gene expression and chromatin accessibility measurements in developing single cells, while preserving cell type variations and cellular dynamics. First, scMultiNODE uses a scalable, Quantized Gromov-Wasserstein optimal transport to align a large number of cells across different measurements. Next, it utilizes neural ordinary differential equations to explicitly model cell development with a regularization term to learn a dynamic latent space. Experiments on six real-world developmental single-cell datasets demonstrate that scMultiNODE can integrate temporally profiled multi-modal single-cell measurements more effectively than existing methods that focus on cell type variations and often overlook cellular dynamics. We also demonstrate that scMultiNODE's joint latent space facilitates several insightful downstream analyses of single-cell development, including the investigation of complex cell trajectories and the enabling of cross-modal label transfer. The data and code are publicly available at https://github.com/rsinghlab/scMultiNODE.

autoencoders↗