Figure 2 from CoMIR: Contrastive Multimodal Image Representation for Registration | Semantic Scholar (2024)

[PDF] Semantic Reader

This work proposes a semantic similarity metric for image registration that learns dataset-specific features that drive the optimization of a learning-based registration model and achieves consistently high registration accuracy.

The results suggest that, although I2I translation may be helpful when the modalities to register are clearly correlated, registration of modalities which express distinctly different properties of the sample are not well handled by the I 2I translation approach.

This work presents INSPIRE, a top-performing general-purpose method for deformable image registration which brings distance measures which combine intensity and spatial information into an elastic B-splines-based transformation model and incorporates an inverse inconsistency penalization supporting symmetric registration performance.

An end-to-end semi-supervised cross-modal contrastive learning framework that simultaneously performs disease classi-fication and localization tasks and constitutes a feedback loop for image and radiomic modality features to mutually reinforce each other, which yields cross-modality representations that are both robust and interpretable.

This thesis develops and improve methods for image segmentation, retrieval and statistical analysis, with applications in imaging-based diagnostic pipelines, and examines a number of methods for multiplicity correction on statistical analyses of correlation using medical images.

This work builds upon previous successes in biological image translation (XAcGAN) and mono-modal image registration (RoTIR) to develop a deep learning model, Dual-Domain RoTIR (DD_RoTIR), specifically designed to address these challenges.

The proposed method demonstrates general applicability and consistently outperforms reference registration tools elastix and VoxelMorph and introduces new equivariance constraints to improve the consistency of CoMIRs under deformation.

The performance drop is investigated by exploiting recent insights in contrastive learning in classification and self-supervised learning and visualize the spatial relations of the learned representations by means of multidimensional scaling, and show that additional supervision on the bottleneck layer can lead to partial dimensional collapse of the intermediate embedding space.

A new application-independent content-based image retrieval (CBIR) system for reverse (sub-)image search across modalities, which combines deep learning to generate representations with robust feature extraction and bag-of-words models for efficient and reliable retrieval is proposed.

...

...

Experimental results demonstrate that the proposed methods can achieve superior performance regarding both accuracy and robustness, which can be used to rigidly register multi-modal images and provide an initial estimation for non-rigid registration in clinical practices.

It is shown that composition of data augmentations plays a critical role in defining effective predictive tasks, and introducing a learnable nonlinear transformation between the representation and the contrastive loss substantially improves the quality of the learned representations, and contrastive learning benefits from larger batch sizes and more training steps compared to supervised learning.

This work develops a model which learns image representations that significantly outperform prior methods on the tasks the authors consider, and extends this model to use mixture-based representations, where segmentation behaviour emerges as a natural side-effect.

A symmetric, intensity interpolation-free, affine registration framework based on a combination of intensity and spatial information is proposed, which exhibits greater robustness and higher accuracy than similarity measures in common use, when inserted into a standard gradient-based registration framework.

A simple addition to the standard SCNN loss function is proposed that encourages the SCNN to be rotationally equivariant, and is easily added to modern SCNNs and achieves improved equivariance and yields performance improvements on average.

This work presents an approach based on disentangled representation for generating diverse outputs without paired training images that can generate diverse and realistic images on a wide range of tasks without pairedTraining data.

The deep neural network is resorts to, and tries to learn descriptors for multimodal image patch matching, which is the key issue of image registration, and shows superiority over other state-of-the-art approaches.

It is shown that structure matters: incorporating knowledge about locality in the input into the objective can significantly improve a representation’s suitability for downstream tasks and is an important step towards flexible formulations of representation learning objectives for specific end-goals.

...

...