This PDF file contains the front matter associated with SPIE Proceedings Volume 11313, including the Title Page, Copyright information, Table of Contents, Author and Conference Committee list

Mixed Reality (MR) headsets have the potential to revolutionize the way we work, learn, communicate, and get entertained. The main pillars for MR development are wearable, visual and social comfort, as well as immersion experience. Do these pillars intersect the specific needs of medical imaging and surgical guidance applications? We will review the various challenges to implement MR hardware specifically adapted to such tasks with today’s start of the art MR headset technology.

We propose a deep learning-based method to perform arbitrary image-to-image translations among four types of MRI scans, including T1-weighted, T1c (T1-weighted with contrast enhancement), Flair and T2-weighted. The goal is to rapidly generate different contrast weighted images which provide comprehensive diagnostic information. The proposed method employs a unified generative adversarial network (unified GAN) which translates any randomly selected MRI scan to the rest scan types. Compared to traditional GAN which takes only images as input, the proposed unified GAN takes both the original image and target domain label as input. The proposed method was evaluated using 50 patients’ brain datasets with well-aligned multi-types of MRI scans. Normalized mean absolute error (NMAE) and peak signal-tonoise ratio (PSNR) were used to quantify the synthesis accuracy of the proposed method. With T2 scan as input, the average NMAE was 0.018±0.003, 0.014±0.002, and 0.022±0.005 for T1, T1c and Flair MRI scans, respectively. The average PSNR was 30.1±3.7 dB, 36.3±3.5 dB, and 30.4±4.7 dB for T1, T1c and Flair MRI scans, respectively. Image quality of the synthesized MRI scans are comparable to original MRI scans.

In this work, we extend the joint-intensity fusion (JIF) algorithm to use normalized cross-correlation (NCC) as the weighting metric for use in the context of cross-modality MRI synthesis, rather than sum squared intensity differences (SSD). We evaluate our method using the Kirby dataset, by synthesizing a representative FLAIR from a target subject’s T1w image. The accuracy of the synthetic FLAIR can be confirmed using the FLAIR taken for the target subject during the imaging session. For each subject in the dataset, using NCC with JIF results in a 51% lower mean absolute error than using SSD.

This paper newly introduces multi-modality loss function for GAN-based super-resolution that can maintain image structure and intensity on unpaired training dataset of clinical CT and micro CT volumes. Precise non- invasive diagnosis of lung cancer mainly utilizes 3D multidetector computed-tomography (CT) data. On the other hand, we can take μCT images of resected lung specimen in 50 μm or higher resolution. However, μCT scanning cannot be applied to living human imaging. For obtaining highly detailed information such as cancer invasion area from pre-operative clinical CT volumes of lung cancer patients, super-resolution (SR) of clinical CT volumes to μCT level might be one of substitutive solutions. While most SR methods require paired low- and high-resolution images for training, it is infeasible to obtain precisely paired clinical CT and μCT volumes. We aim to propose unpaired SR approaches for clincial CT using micro CT images based on unpaired image translation methods such as CycleGAN or UNIT. Since clinical CT and μCT are very different in structure and intensity, direct appliation of GAN-based unpaired image translation methods in super-resolution tends to generate arbitrary images. Aiming to solve this problem, we propose new loss function called multi-modality loss function to maintain the similarity of input images and corresponding output images in super-resolution task. Experimental results demonstrated that the newly proposed loss function made CycleGAN and UNIT to successfully perform SR of clinical CT images of lung cancer patients into μCT level resolution, while original CycleGAN and UNIT failed in super-resolution.

Multimodal computed tomography (CT) scans, including non-contrast CT (NCCT), CT Perfusion (CTP), and CT Angiography (CTA) are widely used in acute stroke diagnosis and treatment planning. While each imaging modality is for different visualization purposes such as anatomical structures and functional information, image quality is obtained variously. In this work, we aim at enhancing the image quality for all modalities by using deep learning technology. Through our experiments, we demonstrate that by using transfer learning and generative adversarial network, NCCT images are beneficial for CTP image reconstruction, and CTP images are helpful for CTA image quality enhancement.

The assistance of deep learning techniques for clinic doctors in disease analysis, diagnosis and treatment is becoming popular and popular. In this paper, we propose a U-shape architecture based Group Attention network (named as GANet) for symptom segmentation in fundus images with diabetic retinopathy, in which Channel Group Attention(CGA) module and Spatial Group Attention Upsampling (SGAU) module are designed. The CGA module can adaptively allocate resources based on the importance of the feature channels, which can enhance the flexibility of the network to handle different types of information. The original U-Net directly merges the high-level features and low-level features in decoder stage for semantic segmentation, and achieves good results. To increase the nonlinearity of the U-shape network and pay more attention to the lesion area, we propose a Spatial Group Attention Upsampling (SGAU) module. In summary, our main contributions include two aspects: (1) Based on the U-shape network, the CGA module and SGAU module are designed and applied, which can adaptively allocate the weight of channels and pay more attention to the lesion area, respectively. (2) Compared with the original U-Net, the Dice coefficients of the proposed network improves by nearly 2.96% for hard exudates segmentation and 2.89% for hemorrhage segmentation, respectively.

Diabetic retinopathy (DR), a highly specific vascular complication caused by diabetes, has been found a major cause of blindness in the world. Early screening of DR is crucial for prevention of vision loss. Hard exudates (HEs) is one of the main manifestations of DR, which is characterized by hyper-reflective foci (HF) in retinal optical coherence tomography(OCT) images. In this paper, a fully automated method based on U-shape network is proposed to segment HF in retinal OCT images. Compared with the original U-Net, there are two main improvements in the proposed network:(1) The ordinary 3×3 convolution is replaced by multi-scale convolution based on dilated convolution, which can achieve adaptive receptive fields of the images. (2) In order to ignore irrelevant information and focus on key information in the channels, the channel attention module is embedded in the model. A dataset consisting of 112 2D OCT B-scan images was used to evaluate the proposed U-shape network for HF segmentation with 4-fold cross validation. The mean and standard deviation of Dice similarity coefficient, recall and precision are 73.26±2.03%, 75.71±1.98% and 74.28± 2.67%, respectively. The experimental results show the effectiveness of the proposed method.

Deep networks provide excellent image segmentation results given copious amounts of supervised training data (source data). However, when a trained network is applied to data acquired at a different clinical center or on a different imaging device (target data), a significant drop in performance can occur due to the domain shift between the test data and the network training data. To solve this problem, unsupervised domain adaptation methods retrain the model with labeled source data and unlabeled target data. In real practice, retraining the model is time consuming and the labeled source data may not be available for people deploying the model. In this paper, we propose a straightforward unsupervised domain adaptation method for multi-device retinal OCT image segmentation which does not require labeled source data and does not require retraining of the segmentation model. The segmentation network is trained with labeled Spectralis images and tested on Cirrus images. The core idea is to use a domain adaptor to convert target domain images (Cirrus) to a domain that can be segmented well by the already trained segmentation network. Unlabeled Spectralis and Cirrus images are used to train this domain adaptor. The domain adaptation block is used before the trained network and a discriminator is used to differentiate the segmentation results from Spectralis and Cirrus. The domain adaptation portion of our network is fully unsupervised and does not change the previously trained segmentation network.

Segmentation of heart substructures in 2D echocardiography images is an important step in diagnosis and management of cardiovascular disease. Given the ubiquity of echocardiography in routine cardiology practice, the time-consuming nature of manual segmentation, and the high degree of inter-observer variability, fully automatic segmentation is a goal common to both clinicians and researchers. The recent publication of the annotated CA- MUS dataset will help catalyze these efforts. In this work we develop and validate against this dataset a deep fully convolutional neural network architecture for the multi-structure segmentation of echocardiography, in- cluding the left ventricular endocardium and epicardium, and the left atrium. In ten-fold cross validation with data augmentation, we obtain mean Dice overlaps of 0.93, 0.95, and 0.89 on the three structures respectively, representing state of the art on this dataset. We further report small biases and narrow limits of agreement between the automatic and manual segmentations in derived clinical indices, including median absolute errors for left ventricular diastolic (7.4mL) and systolic (4.8mL) volumes, and ejection fraction (4.1%), within previously reported inter-observer variability. These encouraging results must still be validated against large-scale independent clinical data.

The choroid is a vascular plexus located between the retina and the sclera, providing oxygen and nourishment to the outer layers of the retina. Thickness changes in the choroid are of importance in the pathophysiology of various ocular diseases such as glaucoma, age-related macular degeneration (AMD), and others. Our previous choroidal layer segmentation method of 3D macular optical coherence tomography (OCT) scans using choroidal vessel segmentation tended to segment thinner choroidal layers than ground truths, requiring long running time and much memory. To overcome these drawbacks, we introduce a new, fast, and memory-efficient multiresolution LOGISMOS (layered optimal graph image segmentation for multiple objects and surfaces) method. The key idea of the method is to consequently segment the choroidal layer in the higher resolution sub-OCT image volume constrained by the layer segmented in the lower resolution OCT image volume to reduce the size of columns for graph search. Generally, it outperformed our previous method and showed a similar performance to the inter-observer variability between 2 experts.

Reducing speckle noise is an important task for improving visual and automated assessment of retinal OCT images. Traditional image/signal processing methods only offer moderate speckle reduction; deep learning methods can be more effective but require substantial training data, which may not be readily available. We present a novel self-fusion method that offers effective speckle reduction comparable to deep learning methods, but without any external training data. We present qualitative and quantitative results in a variety of datasets from fovea and optic nerve head regions, with varying SNR values for input images.

Annual low dose computed tomography (CT) lung screening is currently advised for individuals at high risk of lung cancer (e.g., heavy smokers between 55 and 80 years old). The recommended screening practice significantly reduces all-cause mortality, but the vast majority of screening results are negative for cancer. If patients at very low risk could be identified based on individualized, image-based biomarkers, the health care resources could be more efficiently allocated to higher risk patients and reduce overall exposure to ionizing radiation. In this work, we propose a multi-task (diagnosis and prognosis) deep convolutional neural network to improve the diagnostic accuracy over a baseline model while simultaneously estimating a personalized cancer-free progression time (CFPT). A novel Censored Regression Loss (CRL) is proposed to perform weakly supervised regression so that even single negative screening scans can provide small incremental value. Herein, we study 2287 scans from 1433 de-identified patients from the Vanderbilt Lung Screening Program (VLSP) and the Consortium for Molecular and Cellular Characterization of Screen-Detected Lesions (MCL) cohorts. Using five-fold cross-validation, we train a 3D attention-based network under two scenarios: (1) single-task learning with only classification, and (2) multi-task learning with both classification and regression. The single-task learning leads to a higher AUC compared with the Kaggle challenge winner pre-trained model (0.878 v. 0.856), and multitask learning significantly improves the single-task one (AUC 0.895, p<0.01, McNemar test). In summary, the image based predicted CFPT can be used in follow-up year lung cancer prediction and data assessment.

Fully Convolutional Neural Networks (FCNNs) have been widely employed to solve object segmentation tasks effectively in both the computer vision and medical image processing fields in recent years. In object segmentation, FCNNs play a pixel-level prediction role in generating segmented predictions pixel-by-pixel, but they ignore the relationships among generated pixels on the output image. Moreover, blurry boundaries of predicted objects are another common obstacle in this task, because FCNNs usually generate low-frequency components of an image well, but they lack clear high-frequency information inside. In order to solve these problems, we introduce a top-down strategy by globally considering object shapes and context information. Moreover, based on original pixel-wise loss functions which we call a bottom-up strategy, we formulate the tumor segmentation task as a regression problem by using Jaccard Similarity Coefficient (JSC) which is usually one of the main metrics for evaluating the performance of segmentation methods. We directly propose a JSC loss to further optimize network parameters for globally evaluating the whole outputs of tumors. Furthermore, the new loss also alleviates the effect of severe class imbalanced problems between tumor regions and non-tumor regions when training FCNNs. By leveraging the bottom-up and top-down strategies together, our model can obtain more fine-grained tumor segmentation results and also be easily embedded into any FCNN framework for other object segmentation tasks. Detailed experimental results on common metrics demonstrate the superior performance of our proposed method for Kidney Tumor Segmentation Challenge 2019 among more than 100 involved teams.

Multiple instance learning (MIL) is a supervised learning methodology that aims to allow models to learn instance class labels from bag class labels, where a bag is defined to contain multiple instances. MIL is gaining traction for learning from weak labels but has not been widely applied to 3D medical imaging. MIL is well-suited to clinical CT acquisitions since (1) the highly anisotropic voxels hinder application of traditional 3D networks and (2) patch-based networks have limited ability to learn whole volume labels. In this work, we apply MIL with a deep convolutional neural network to identify whether clinical CT head image volumes possess one or more large hemorrhages (>; 20cm³ ), resulting in a learned 2D model without the need for 2D slice annotations. Individual image volumes are considered separate bags, and the slices in each volume are instances. Such a framework sets the stage for incorporating information obtained in clinical reports to help train a 2D segmentation approach. Within this context, we evaluate the data requirements to enable generalization of MIL by varying the amount of training data. Our results show that a training size of at least 400 patient image volumes was needed to achieve accurate per-slice hemorrhage detection. Over a five-fold cross-validation, the leading model, which made use of the maximum number of training volumes, had an average true positive rate of 98.10%, an average true negative rate of 99.36%, and an average precision of 0.9698. The models have been made available along with source code¹ to enabled continued exploration and adaption of MIL in CT neuroimaging.

Coronary artery calcification (CAC) is a strong and independent predictor of cardiovascular disease (CVD) that can be quantified in CT scans showing the heart. CAC lesions are defined as lesions in the coronary arteries with image intensity above 130 HU. The use of a threshold may lead to under- or over-estimation of the amount of CAC and, hence, to incorrect cardiovascular categorization of patients. This is especially pronounced in CT scans without ECG-synchronization where lesions are more subject to cardiac motion and partial volume effects. To address this, we propose a method for quantification of CAC without a threshold. A set of 373 cardiac and 1181 chest CT scans was included to develop the method and a set of 21 scan-rescan pairs (42 scans) was included for final evaluation. Assuming that the attenuation of CAC is superimposed on the attenuation of the artery, we aimed to separate the CAC from the coronary arteries by employing a CycleGAN to generate a synthetic image without CAC from an image containing CAC and vice versa. By subtracting the synthetic image without CAC from the image with CAC, a CAC map is created. The CAC-map can subsequently be used to identify and quantify CAC. The ground truth, i.e. the true amount of CAC, can not be established, therefore, in this work the results generated by the method are compared with clinical calcium scoring in terms of reproducibility. The average relative difference between the calcium scores in scan-rescan pairs of scans was 50% with the proposed method and 86% for the conventional method. Moreover, the correlation between CAC pseudo masses in scan-rescan pairs was 0.92 with the proposed method and 0.89 with conventional calcium scoring. Our proposed method is able to identify and quantify CAC lesions in CT scans without using an intensity level thresholding. This might allow for more reproducible quantification of CAC in CT scans made without ECG synchronization, and, therefore, it might allow more accurate CVD risk prediction.

Medical images are often used to detect and characterize pathology and disease; however, automatically identifying and segmenting pathology in medical images is challenging because the appearance of pathology across diseases varies widely. To address this challenge, we propose a Bayesian deep learning method that learns to translate healthy computed tomography images to magnetic resonance images and simultaneously calculates voxel-wise uncertainty. Since high uncertainty occurs in pathological regions of the image, this uncertainty can be used for unsupervised anomaly segmentation. We show encouraging experimental results on an unsupervised anomaly segmentation task by combining two types of uncertainty into a novel quantity we call scibilic uncertainty.

In CT colonography (CTC), residual stool is a possible confounder in the detection of colonic polyps. While there is a clear clinical need for reduced or minimal bowel preparation for CT colonography, residual stool that is poorly tagged by oral contrast agent prevents satisfactory electronic cleansing (EC) by standard methods on conventional CT. Our study aims to answer quantitatively whether dual-layer spectral-CT allows superior discrimination of residual stool. 60 spectral CT colonography scans were obtained in clinical practice, and careful exhaustive ground truth was established by consensus reading.

Results indicate that spectral CT adds significant discrimination power, in particular when utilizing local spectral variances and covariances, which can be computed efficiently by standard Gaussian filter operations. Simple linear spectral material separation, however, is sufficient only in extended homogeneous regions. In subtle finely structured transition areas, non-linear classifiers or convolutional neural networks are required because of non-linear local multi material superposition effects.

This study developed the estimation method of the liver elasticity using the finite element method (FEM) based on the four-dimensional computed tomography (4DCT) images acquired for radiotherapy planning, and to evaluate the feasibility of estimated elasticity as a biomarker for diagnose liver fibrosis. Fifteen patients who underwent 4DCT images and gadoxetate-acid (Gd-EOB-DTPA)-enhanced magnetic resonance imaging (MRI) were enrolled in this study. The displacement vector fields were calculated between 4DCT images at the end-inspiration and at the end-exhalation using deformable image registration (actual respiratory-induced displacement). Further, we simulated the displacement during respiration by using the FEM (simulated respiratory-induced displacement). The elasticity in each element of liver model was optimized minimalizing the error between actual and simulated respiratory-induced displacement. In Gd-EOB-DTPAenhanced MRI, liver-to-spleen signal intensity ratio (LSR) was calculated using the mean signal intensity for whole liver and spleen. The correlations with two serum biomarkers (APRI: aspartate-aminotransferase to platelet ratio index, FIB-4: Fibrosis-4 index) for elasticity and for LSR were evaluated. The elasticity were strong correlation with APRI-score (r= 0.82), and with FIB-4-score (r= 0.86). On the other hand, LSR were modelate correlation with APRI-score (r= 0.32), and with FIB-4-score (r= 0.30). The mean ± standard deviation of errors between actual and simulated respiratory-induced displacement in the liver model was 0.63 ± 0.41 mm. In this study, liver elasticity was estimated using the FEM and respiratory-induced liver motion obtained from 4DCT images. Furthermore, the estimated elasticity could be a feasible imaging biomarker for diagnose the various degrees of liver fibrosis.

Predicting the future occurrence of Alzheimer’s disease (AD) in patients with mild cognitive impairment (MCI) is a topic of active research. Many papers have formulated this question as a classification problem: one considers a fixed time of conversion and aims to discriminate between the patients who have converted to AD at that time and those who have not. However, a clinically more relevant question is to predict the date at which a patient to AD. Survival analysis is an adequate statistical framework for such a task. Multimodal data (imaging and genetic) provide complementary information for the prediction. While imaging data provides an estimate of the current patient’s state, genetic variants can be associated to the speed of progression to AD. Although they do not provide the same type of information, most papers in classification or regression put imaging and genetic variables on the same level in order to predict the current or future patient’s state. In this work, we propose a survival model using multimodal data to estimate the conversion date to AD, by considering joint effects between the imaging and genetic modalities. We introduce an adapted penalty in the survival model, the group lasso penalty, over joint groups of genes and brain regions. The model is evaluated on genetic (single nucleotide polymorphisms) and imaging (anatomical MRI measures) data from the ADNI database, and compared to a standard Cox model.

Generalizability is an important problem in deep neural networks, especially in the context of the variability of data acquisition in clinical magnetic resonance imaging (MRI). Recently, the Spatially Localized Atlas Network Tiles (SLANT) approach has been shown to effectively segment whole brain non-contrast T1w MRI with 132 volumetric labels. Enhancing generalizability of SLANT would enable broader application of volumetric assessment in multi-site studies. Transfer learning (TL) is commonly to update neural network weights for local factors; yet, it is commonly recognized to risk degradation of performance on the original validation/test cohorts. Here, we explore TL by data augmentation to address these concerns in the context of adapting SLANT to anatomical variation (e.g., adults versus children) and scanning protocol (e.g., non-contrast research T1w MRI versus contrast-enhanced clinical T1w MRI). We consider two datasets: First, 30 T1w MRI of young children with manually corrected volumetric labels, and accuracy of automated segmentation defined relative to the manually provided truth. Second, 36 paired datasets of pre- and post-contrast clinically acquired T1w MRI, and accuracy of the post-contrast segmentations assessed relative to the pre-contrast automated assessment. For both studies, we augment the original TL step of SLANT with either only the new data or with both original and new data. Over baseline SLANT, both approaches yielded significantly improved performance (pediatric: 0.89 vs. 0.82 DSC, p<0.001; contrast: 0.80 vs 0.76, p<0.001 ). The performance on the original test set decreased with the new-data only transfer learning approach, so data augmentation was superior to strict transfer learning.

Arterial Spin Labeling (ASL) is a popular non-invasive neuroimaging technique to use MRI for quantitatively Cerebral Blood Flow (CBF) mapping. However, ASL usually suffers from poor signal quality and repeated measurements are typically acquired to improve signal quality through averaging at the cost of long scan time. In this work, a deep learning algorithm is proposed to leverage both convolutional neural network (CNN) based image enhancement as well as combining complementary/mutual information from multiple tissue contrasts in ASL acquisition. Both quantitative and qualitative evaluation demonstrate the performance and stability of the proposed algorithm and its superiority over conventional denoising algorithms and standard deep learning based denoising. The results demonstrate the feasibility of efficient and high-quality ASL measurements from average-free fast acquisition which will enable broader clinical application of ASL.

Motion artifacts in MRI degrade image quality typically with blurring or ghosting across the image along the phase encoding direction, and severe motion artifacts may result in non-diagnostic exams. We developed a deep learning model based on a densely connected residual network (DRN) with K-space blending (DRN-KB) method in a purpose to reduce motion artifacts in brain MRI. Our DRN model took advantage of residual learning and dense connection to achieve higher performance in motion reduction compared with denoising convolutional neural network (DnCNN). In addition, to overcome over-smoothing and reduced tissue contrast in the motion-reduction images produced by DRN, K-space blending was performed that part of central K-space of the original motion image were reserved whereas the remaining Kspace was replaced by the motion-reduction images from DRN. The final MRI images were reconstructed using the blended K-space. The optimal blending ratio of the K-space was determined during the training process. Our DRN model was trained and tested with the axial T1-weighted (T1W) images with simulated motion. Two clinical cases (50 images) were used in training and validation and 16 cases (417 images) were used in testing. Structural SIMilarity (SSIM) index and improvement in signal-to-noise ratio (ISNR) were calculated to evaluate the image quality. Our DRN-KB method reduced motion artifacts with an increased SSIM of motion-reduction images (SSIM:0.95) compared with that of original motion images (SSIM:0.83) and demonstrated an improved SNR (ISNR: 2.89 dB). The performance of DRN-KB method was significantly better than that of the conventional DnCNN (P-value < 0.05).

Longitudinal brain alignment is critical for disease monitoring and adaptive treatment planning in glioblastoma (GBM) patients. However, the current methods are either non-adaptive to pathological brains, or time and laborintensive. Here, we aim to develop a novel deep-learning-based framework for longitudinal postoperative brain GBM scan registration. The proposed pathology adaptive registration framework (PARF) adopts a double UNET architecture: a 2D 7-level UNET, NET_seg, for pathology segmentation, and a 3D 5-level UNET, NET_reg, for unsupervised image registration, connected through a spatial transformer and a volume combiner. NET_seg was first trained separately and then combined with NET_reg for pathology adaptive registration training. In aggregated registration testing of PARF, 36 registrations from 18 intra-subject pairs of post-operative follow-up MR scans were selected, and the results were compared to those from current state-of-the-art methods as well as non-adaptive NET_reg alone. PARF is significantly faster and more accurate than comparison methods, in terms of sum-of-squared differences, segmentation alignment dice coefficients, and landmark mislignment errors. PARF may pave the path for various clinical and research applications that depend on the accurate registration of GBM longitudinal images.

Adaptive radiotherapy is an effective procedure for the treatment of cancer, where the daily anatomical changes in the patient are quantified, and the dose delivered to the tumor is adapted accordingly. Deformable Image Registration (DIR) inaccuracies and delays in retrieving and registering on-board cone beam CT (CBCT) image datasets from the treatment system with the planning kilo Voltage CT (kVCT) have limited the adaptive workflow to a limited number of patients. In this paper, we present an approach for improving the DIR accuracy using a machine learning approach coupled with biomechanically guided validation. For a given set of 11 planning prostate kVCT datasets and their segmented contours, we first assembled a biomechanical model to generate synthetic abdominal motions, bladder volume changes, and physiological regression. For each of the synthetic CT datasets, we then injected noise and artifacts in the images using a novel procedure in order to mimic closely CBCT datasets. We then considered the simulated CBCT images for training neural networks that predicted the noise and artifact-removed CT images. For this purpose, we employed a constrained generative adversarial neural network, which consisted of two deep neural networks, a generator and a discriminator. The generator produced the artifact-removed CT images while the discriminator computed the accuracy. The deformable image registration (DIR) results were finally validated using the model-generated landmarks. Results showed that the artifact-removed CT matched closely to the planning CT. Comparisons were performed using the image similarity metrics, and a normalized cross correlation of >0.95 was obtained from the cGAN based image enhancement. In addition, when DIR was performed, the landmarks matched within 1.1 +/- 0.5 mm. This demonstrates that using an adversarial DNN-based CBCT enhancement, improved DIR accuracy bolsters adaptive radiotherapy workflow.

While deep learning based methods for medical deformable image registration have recently shown significant advances in both speed and accuracy, methods for use in radio therapy are still rarely proposed due to several challenges such as low contrast and artifacts in cone beam CT (CBCT) images or extreme deformations. The aim of image registration in radio therapy is to align a baseline CT and low-dose CBCT images, which allows contours to be propagated and applied doses to be tracked over time. To this end, we present a novel deep learning method for multi-modal deformable CT-CBCT registration. We train a CNN in weakly supervised manner, aiming to optimize an edge-based image similarity and a deformation regularizer including a penalty for local changes of topology and foldings. Additionally, we measure the alignment of given segmentations, facing the problem of extreme deformations. Our method receives only CT and a CBCT images as input and uses groundtruth segmentations exclusively during training. Furthermore, our method is not dependent on the availability of difficult to access ground-truth deformation vector fields. We train and evaluate our method on follow-up image pairs of the pelvis and compare our results to conventional iterative registration algorithms. Our experiments show that the registration accuracy of our deep learning based approach is superior to iterative registration without additional guidance by segmentations and nearly as good as iterative structure guided registration that requires ground-truth segmentations. Furthermore, our deep learning based method runs approximately 100 times faster than the iterative methods.

Current unsupervised deep learning-based image registration methods are trained with mean squares or nor- malized cross correlation as a similarity metric. These metrics are suitable for registration of images where a linear relation between image intensities exists. When such a relation is absent knowledge from conventional image registration literature suggests the use of mutual information. In this work we investigate whether mutual information can be used as a loss for unsupervised deep learning image registration by evaluating it on two datasets: breast dynamic contrast-enhanced MR and cardiac MR images. The results show that training with mutual information as a loss gives on par performance compared with conventional image registration in contrast enhanced images, and the results show that it is generally applicable since it has on par performance compared with normalized cross correlation in single-modality registration.

Diffusion-weighted magnetic resonance imaging (DW-MRI) is the only non-invasive approach for estimation of intravoxel tissue microarchitecture and reconstruction of in vivo neural pathways for the human brain. With improvement in accelerated MRI acquisition technologies, DW-MRI protocols that make use of multiple levels of diffusion sensitization have gained popularity. A well-known advanced method for reconstruction of white matter microstructure that uses multishell data is multi-tissue constrained spherical deconvolution (MT-CSD). MT-CSD substantially improves the resolution of intra-voxel structure over the traditional single shell version, constrained spherical deconvolution (CSD). Herein, we explore the possibility of using deep learning on single shell data (using the b=1000 s/mm² from the Human Connectome Project (HCP)) to estimate the information content captured by 8^th order MT-CSD using the full three shell data (b=1000, 2000, and 3000 s/mm² from HCP). Briefly, we examine two network architectures: 1.) Sequential network of fully connected dense layers with a residual block in the middle (ResDNN), 2.) Patch based convolutional neural network with a residual block (ResCNN). For both networks an additional output block for estimation of voxel fraction was used with a modified loss function. Each approach was compared against the baseline of using MT-CSD on all data on 15 subjects from the HCP divided into 5 training, 2 validation, and 8 testing subjects with a total of 6.7 million voxels. The fiber orientation distribution function (fODF) can be recovered with high correlation (0.77 vs 0.74 and 0.65) and low root mean squared error ResCNN:0.0124, ResDNN:0.0168 and sCSD:0.0323 as compared to the ground truth of MT-CST, which was derived from the multi-shell DW-MRI acquisitions. The mean squared error between the MT-CSD estimates for white matter tissue fraction and for the predictions are ResCNN:0.0249 vs ResDNN:0.0264. We illustrate the applicability of high definition fiber tractography on a single testing subject with arcuate and corpus callosum Tractography. In summary, the proposed approach provides a promising framework to estimate MT-CSD with limited single shell data. Source code and models have been made publicly available.

The application of deep learning to build accurate predictive models from functional neuroimaging data is often hindered by limited dataset sizes. Though data augmentation can help mitigate such training obstacles, most data augmentation methods have been developed for natural images as in computer vision tasks such as CIFAR, not for medical images. This work helps to fills in this gap by proposing a method for generating new functional Magnetic Resonance Images (fMRI) with realistic brain morphology. This method is tested on a challenging task of predicting antidepressant treatment response from pre-treatment task-based fMRI and demonstrates a 26% improvement in performance in predicting response using augmented images. This improvement compares favorably to state-of-the-art augmentation methods for natural images. Through an ablative test, augmentation is also shown to substantively improve performance when applied before hyperparameter optimization. These results suggest the optimal order of operations and support the role of data augmentation method for improving predictive performance in tasks using fMRI.

Schizophrenia is associated with cognitive impairments. Exercise interventions including yoga and aerobic exercise have been shown to improve cognitive functioning in schizophrenic patients. Yet the underlying neural basis is not clear for this cognitive improvement. This work aimed to investigate the brain functional effect caused by exercise interventon in female patients with early psychosis. Resting-state fMRI was collected longitudinally from 71 patients who were randomized into three programs: yoga, aerobic exercise and waitlist control. Functional connectivity matrices of each individual at baseline and 12-week follow-up timepoint were estimated. Then the connectivity changes were calculated and used as potential predictors to classify the three groups. A machine learning method gcForest was used to train classification models on a subset and tested on the rest of the data. Classification performance was evaluated using multiple n-fold cross-validation to ensure a robust estimate of the accuracy. The classification accuracy ranges from 86.31 to 94.00. The most predictive features were examined in the brain, which include connectivity changes along several major pathways in high order functional networks including default mode and executive control networks. This is the first study showing that the connectivity alterations can successfully distinguish intervention from control groups, and also detect the two different types of intervention: yoga and aerobic exercise. Findings suggest that the altered functional connectivity may contribute to the cognitive improvement after intervention. Our work sheds light on the use of advanced neuroimaging and machine learning approaches to explore potential biomarkers for predicting outcomes of exercise intervention in psychosis.

An important goal in neuroscience is to understand the relationship between brain activity and cognitive traits. Toward this aim, many studies draw upon resting-state Functional Magnetic Resonance Imaging (rs-fMRI) datasets, which provide a means of probing the spatial and temporal structure of spontaneous brain activity in human subjects. However, as rsfMRI and behavioral data are both noisy, obtaining a robust relationship between them is difficult. Further, given the large number of candidate features in fMRI data, it is challenging to select those which may be most relevant for predicting a specific behavioral trait. In our research, we examined brain fMRI features based upon Sample Entropy (SampEn), which is a nonlinear signal processing measure that captures the complexity of a time series. Using 90 selected regions of interest (ROIs) across 96 unrelated subjects from the Human Connectome Project (HCP), we found that our SampEn-based features contained reproducible patterns over different rs-fMRI scans. Further, we report the relative stability of each ROI’s SampEn over four different scans of these 96 subjects. Finally, we apply multivariate models to relate SampEnbased brain features to cognition and emotion-related behavioral measures, and show that these models are reproducible when applied to different scans from the same individuals. Overall, these results suggest that SampEn of regional fMRI signals may be a reproducible metric of brain activity in healthy subjects.

Clinical applications of quantitative diffusion-weighted imaging (qDWI) require confidence intervals for derived diffusion parameters to aid differentiation of technical errors from tissue characteristics. This study outlines practical procedures to evaluate precision (uncertainty) and accuracy (bias) of parametric maps derived for non-Gaussian diffusion models using numerical DWI phantoms (digital reference objects (DROs)) generated for advanced qDWI clinical trial protocols. The generated DROs include simulated acquisition noise, DICOM scaling and clinically-relevant qDWI parameter ranges for perfusion-fraction intra-voxel incoherent motion, kurtosis and stretched exponential diffusion models. Evaluation of fit accuracy and precision is illustrated for unsupervised linear least-squares (LLS) versus nonlinear minimization (NLM) algorithms ignorant of noise. DRO application examples are shown for calibration of noise-induced uncertainty with a physical DWI phantom, retrospective model fidelity analysis for brain tumor data and prospective error mapping in renal tissue. Physical DWI phantom analysis confirms adequate DRO noise model and realistic predicted fit errors. Detection of step-like bias activation in fit parameter space, consistent with modeldependent signal truncation at the noise floor, is proposed based on differences in NLM versus LLS fit maps. For all diffusion models, noise-induced bias in diffusivity is anti-correlated to bias in the model-specific (non-diffusivity) parameter. For confident assessment of fit parameter precision, bias minimization through b-range constraints and bdependent averaging is advised prior to fit uncertainty mapping. For low bias, higher precision is achieved by LLS versus NLM, and for diffusivity versus model parameter. The described procedures allow efficient qDWI protocol optimization toward reduced acquisition and model-dependent errors.

Deep learning approaches have been used extensively for medical image segmentation tasks. Training deep networks for segmentation, however, typically requires manually delineated examples which provide a ground truth for optimization of the network. In this work, we present a neural network architecture that segments vascular structures in retinal OCTA images without the need of direct supervision. Instead, we propose a variational intensity cross channel encoder that finds vessel masks by exploiting the common underlying structure shared by two OCTA images of the the same region but acquired on different devices. Experimental results demonstrate significant improvement over three existing methods that are commonly used.

We developed an approach for cardiac cine magnetic resonance image (MRI) left ventricle segmentation using small training dataset with noisy manual annotations. Our approach combined the strengths of deep neural networks and normalized cut with continuous regularization. We entered U-net coarse segmentation into the regularized normalized cut module that evaluates the partitioning quality within and between segmentation regions. The resulting challenging optimization problem was efficiently solved through upper bound relaxation in an iterative manner with guaranteed convergence. Within each iteration, we derived an upper bound of the high-order normalized cut term, which was combined with image-grid continuous regularization and solved using a continuous min-cut/max- ow framework. Using 5 and 10 subjects with noisy manual labels for network training, we observed much improved segmentation accuracy and minimized effects due to the size of training dataset and the quality of training annotations.

Dynamic contrast enhanced computed tomography (CT) is an imaging technique that provides critical information on the relationship of vascular structure and dynamics in the context of underlying anatomy. A key challenge for image processing with contrast enhanced CT is that phase discrepancies are latent in different tissues due to contrast protocols, vascular dynamics, and metabolism variance. Previous studies with deep learning frameworks have been proposed for classifying contrast enhancement with networks inspired by computer vision. Here, we revisit the challenge in the context of whole abdomen contrast enhanced CTs. To capture and compensate for the complex contrast changes, we propose a novel discriminator in the form of a multi-domain disentangled representation learning network. The goal of this network is to learn an intermediate representation that separates contrast enhancement from anatomy and enables classification of images with varying contrast time. Briefly, our unpaired contrast disentangling GAN(CD-GAN) Discriminator follows the ResNet architecture to classify a CT scan from different enhancement phases. To evaluate the approach, we trained the enhancement phase classifier on 21060 slices from two clinical cohorts of 230 subjects. The scans were manually labeled with three independent enhancement phases (non-contrast, portal venous and delayed). Testing was performed on 9100 slices from 30 independent subjects who had been imaged with CT scans from all contrast phases. Performance was quantified in terms of the multi-class normalized confusion matrix. The proposed network significantly improved correspondence over baseline UNet, ResNet50 and StarGAN’s performance of accuracy scores 0.54. 0.55, 0.62 and 0.91, respectively (p-value<0.0001 paired t-test for ResNet versus CD-GAN). The proposed discriminator from the disentangled network presents a promising technique that may allow deeper modeling of dynamic imaging against patient specific anatomies.

Vessel wall volume (VWV) and local vessel-wall-plus-plaque thickness (VWT) measured from 3D ultrasound (3DUS) are sensitive to change of plaque burden over time and are useful in evaluating treatment effect. Segmentation of the media-adventitia (MAB) and lumen-intima boundaries (LIB) was required in VWV and VWT quantification. Manual segmentation of these boundaries is time-consuming and prone to observer variability. In this work, we developed and validated a method to segment MAB and LIB from axial images re-sliced from 3DUS images using a light-weight coarse-to-fine network. The proposed network is computationally efficient with only 0.59M parameters (compared to 31M parameters in U-Net). The boundaries segmented by the proposed algorithm were compared with manually segmented boundaries. The proposed algorithm attained Dice similarity coefficients (DSC) of 92:5±3:09% and 85:4±6:04% for MAB and LIB respectively, which are higher than those attained by U-Net family networks, including U-Net++, scaled U-Net and attention U-Net. This segmentation tool will facilitate efficient quantification of VWV and VWT, thereby making it more feasible for them to be measured in clinical trials evaluating treatment effect or for stroke risk stratification.

Chronic obstructive pulmonary disease (COPD) is a common inflammatory disease associated with restricted lung airflow. Quantitative computed tomography (CT)-based bronchial measures are popularly used in COPD-related studies, which require both airway segmentation and anatomical branch labeling. This paper presents an algorithm for anatomical labeling of human airway tree branches using a novel two-step machine learning and hierarchical features. Anatomical labeling of airway branches allows standardized spatial referencing of airway phenotypes in large population-based studies. State-ofthe-art anatomical labeling methods are associated with mandatory manual reviewing and correction for mislabeled branches—a time-consuming process susceptible to inter-observer variability. The new method is fully automated, and it uses hierarchical branch-level features from the current as well as ancestral and descendant branches. During the first machine learning step, it differentiates candidate anatomical branches from insignificant topological branches, often, responsible for variations in airway branching patterns. The second step is designed for lung lobe-based classification of anatomical labels for valid candidate branches. The machine learning classifiers has been designed, trained, and validated using total lung capacity (TLC) CT scans (n = 350) from the Iowa cohort of the nationwide COPDGene study during their baseline visits. One hundred TLC CT scans were used for training and validation, and a different set of 250 scans were used for testing and evaluative experiments. The new method achieved labeling accuracies of 98.4, 97.2, 92.3, 93.4, and 94.1% in the right upper, right middle, right lower, left upper, and left lower lobe, respectively, and an overall accuracy of 95.9%. For five clinically significant segmental branches, the method has achieved an accuracy of 95.2%.

Computer-assisted image segmentation techniques could help clinicians to perform the border delineation task faster with lower inter-observer variability. Recently, convolutional neural networks (CNNs) are widely used for automatic image segmentation. In this study, we used a technique to involve observer inputs for supervising CNNs to improve the accuracy of the segmentation performance. We added a set of sparse surface points as an additional input to supervise the CNNs for more accurate image segmentation. We tested our technique by applying minimal interactions to supervise the networks for segmentation of the prostate on magnetic resonance images. We used U-Net and a new network architecture that was based on U-Net (dual-input path [DIP] U-Net), and showed that our supervising technique could significantly increase the segmentation accuracy of both networks as compared to fully automatic segmentation using U-Net. We also showed DIP U-Net outperformed U-Net for supervised image segmentation. We compared our results to the measured inter-expert observer difference in manual segmentation. This comparison suggests that applying about 15 to 20 selected surface points can achieve a performance comparable to manual segmentation.

The quality of the segmentation of organs and pathological tissues has significantly improved in recent years by using deep learning approaches, which are typically trained with fully supervised learning. For these fully supervised learning methods, an immense amount of fully labeled training data is required, which are, especially in medicine, costly to generate. To overcome this issue, weakly supervised training methods are used, because they do not need fully labeled ground truth data. For the localization of objects weakly supervised learning has already become more important. Recently, weakly supervised learning also became increasingly important in the area of segmentation of pathological tissues. However, these currently available approaches still require additional anatomical information. In this paper, we present a weakly supervised segmentation method that does not need ground truth segmentations as input or additional anatomical information. Our method consists of three classification networks in sagittal, axial, and coronal direction that decide whether a slice contains the structure to be segmented. Then, we use the class activation maps of the classification output to generate a combined segmentation. Our network was trained for the challenging task of pancreas segmentation with the publicly available TCIA pancreas dataset and we reached Dice scores for slices of up to 0.86 and an overall Dice score of up to 0.53.

Gastroenterologists are estimated to misdiagnose up to 25% of esophageal adenocarcinomas in Barrett's Esophagus patients. This prompts the need for more sensitive and objective tools to aid clinicians with lesion detection. Artificial Intelligence (AI) can make examinations more objective and will therefore help to mitigate the observer dependency. Since these models are trained with good-quality endoscopic video frames to attain high efficacy, high-quality images are also needed for inference. Therefore, we aim to develop a framework that is able to distinguish good image quality by a-priori informativeness classification which leads to high inference robustness. We show that we can maintain informativeness over the temporal domain using recurrent neural networks, yielding a higher performance on non-informativeness detection compared to classifying individual images. Furthermore, it is also found that by using Gradient weighted Class Activation Map (Grad-CAM), we can better localize informativeness within a frame. We have developed a customized Resnet18 feature extractor with 3 classifiers, consisting of a Fully-Connected (FC), Long-Short-Term-Memory (LSTM) and a Gated-Recurrent-Unit (GRU) classifier. Experimental results are based on 4,349 frames from 20 pullback videos of the esophagus. Our results demonstrate that the algorithm achieves comparative performance with the current state-of-the-art. The FC and LSTM classifier reach an F1 score of 91% and 91%. We found that the LSTM classifier based Grad-CAMs represent the origin of non-informativeness the best as 85% of the images were found to be highlighting the correct area.

The benefit of our novel implementation for endoscopic informativeness classification is that it is trained end- to-end, incorporates the spatiotemporal domain in the decision making for robustness, and makes the model decisions of the model insightful with the use of Grad-CAMs.

The purpose of this paper is to present multi-organ segmentation method using spatial information-embedded fully convolutional networks (FCNs). Semantic segmentation of major anatomical structure from CT volumes is promising to apply in clinical work ows. A multitude of deep-learning-based approaches have been proposed for 3D image processing. With the rapid development of FCNs, the encoder-decoder network architecture is proved to achieved acceptable performance on segmentation tasks. However, it is hard to obtain the spatial information from sub-volumes during training. In this paper, we extend the spatial position information-embeded FCNs which designed for binary segmentation tor multi-class organ segmentation. We introduced gamma correction in data augmentation to improve the FCNs robustness. We compared the FCNs performance with different normalization methods, including batch normalization and instance normalization. Experiment results showed that our modifications positively influence the segmentation performance on abdominal CT dataset. Our highest average dice score achieves 87.2%, while the previous method achieved 86.2%.

Medical image segmentation remains a difficult, time-consuming task; currently, liver segmentation from abdominal CT scans is often done by hand, requiring too much time to construct patient-specific treatment models for hepatocellular carcinoma. Image segmentation techniques, such as level set methods and convolutional neural networks (CNN), rely on a series of convolutions and nonlinearities to construct image features: neural networks that use strictly mean-zero finite difference stencils as convolution kernels can be treated as upwind discretizations of differential equations. If this relationship can be made explicit, one gains the ability to analyze CNN using the language of numerical analysis, thereby providing a well-established framework for proving properties such as stability and approximation accuracy. We test this relationship by constructing a level set network, a type of CNN whose architecture describes the expansion of level sets; forward-propagation through a level set network is equivalent to solving the level set equation; the level set network achieves comparable segmentation accuracy to solving the level set equation, while not obtaining the accuracy of a common CNN architecture. We therefore analyze which convolution filters are present in a standard CNN, to see whether finite difference stencils are learned during training; we observe certain patterns that form at certain layers in the network, where the learned CNN kernels depart from known convolution kernels used to solve the level set equation.

Symmetric design of encoder-decoder networks is common in deep learning. For almost all segmentation problems, the output segmentation is vastly less complex compared to the input image. However, the effect of the size of the decoder on segmentation performance has not been investigated in literature. This work investigates the effect of reducing decoder size on binary segmentation performance in a medical imaging application. To this end, we propose a methodology to reduce the size of the decoder in encoder-decoder networks, where residual skip connections are employed in combination with a 1x1 convolution instead of concatenations (as employed by U-Net) to achieve models with asymmetric design. The results on the ISIC2017 data set show that the amount of trainable parameters in the decoder can be reduced by up to a factor 100 compared to standard U-Net, while retaining segmentation performance. Additionally, the reduced amount of trainable decoder parameters in the proposed models leads to inference times up to 3 times faster compared to standard U-Net.

Multi-scale contextual information is effective for pixel-level label prediction, i.e. image segmentation. However, such important information is only partially exploited in the existing methods. In this paper, we propose a new network architecture for unified multi-scale feature abstraction. The proposed network performs multi-scale analysis to the input image by using spatial pyramid pooling to obtain scene context information and abstract multi-scale features hierarchically. In addition, we present a new skip pathways to learn context information by fusing semantically similar features and develop a deep supervision mechanism for outputs in different scales. The proposed mechanisms relieve the gradient vanishing problem and enforce semantic feature learning. We extensively evaluated our method on the MICCAI 2017 Liver Tumor Segmentation (LiTS) Challenge Dataset and demonstrate highly competitive performance with single step operation and lightweight 2D networks.

One of the recent developments in deep learning is the ability to train extremely deep residual neural networks, knowing that each residual block produces only marginal changes to the data. The accumulation of these changes result in the network’s improved performance, analogous to a complex but trainable iterative algorithm. This intuition can be merged with the underlying theory of probabilistic graphical models in which these iterative algorithms are common and share the underlying probabilistic and information theoretic basis as deep learning. Prior models have been proposed with limitations on the number of iterations allowed in the solution algorithm due to the linear memory growth during the training process. This paper presents a structured activation layer which implements a conditional random field along with an arbitrary iteration message-passing marginal probability estimation algorithm which requires constant, rather than linear, memory with respect to the number of iterations. In this activation layer, the segmentation labels can be specified hierarchically, incorporating a level of abstract structure and some basic geometrical knowledge directly and easily into the network. Thus, this layer allows for the separation of abstract knowledge brought in by the network designer (in the form of the hierarchical structure) from probabilistic priors learned by the neural network. Preliminary results comparing this activation function to softmax and a similar non-hierarchical activation function indicate that it significantly improves performance in segmentation problems.

There has recently been great progress in automatic segmentation of medical images with deep learning algorithms. In most works observer variation is acknowledged to be a problem as it makes training data heterogeneous but so far no attempts have been made to explicitly capture this variation. Here, we propose an approach capable of mimicking different styles of segmentation, which potentially can improve quality and clinical acceptance of automatic segmentation methods. In this work, instead of training one neural network on all available data, we train several neural networks on subgroups of data belonging to different segmentation variations separately. Because a priori it may be unclear what styles of segmentation exist in the data and because different styles do not necessarily map one-on-one to different observers, the subgroups should be automatically determined. We achieve this by searching for the best data partition with a genetic algorithm. Therefore, each network can learn a specific style of segmentation from grouped training data. We provide proof of principle results for open-sourced prostate segmentation MRI data with simulated observer variations. Our approach provides an improvement of up to 23% (depending on simulated variations) in terms of Dice and surface Dice coefficients compared to one network trained on all data.

Automatic instance segmentation of individual vertebrae from 3D CT is essential for various applications in orthopedics, neurology, and oncology. In case model-based segmentation (MBS) shall be used to generate a mesh-based representation of the spine, a good initialization of MBS is crucial to avoid wrong vertebra labels due to the similar appearance of adjacent vertebrae. Here, we propose to use deep learning (DL) for MBS initialization and for robustly guiding MBS during segmentation to generate 24 instance segmentations for each and every vertebra. We propose a four-step approach: In step 1, we apply a first single-class U-Net to coarsely segment the spine. In step 2, we sample image patches along the coarse segmentation of step 1 and apply a second multi-class U-net to generate a fine segmentation including individual labeling of some key vertebrae and vertebra body landmarks. In step 3, we detect and label landmark coordinates from the classes estimated in step 2. In step 4, we initialize all MBS vertebrae models using the landmarks from step 3 and adapt the model to the joint vertebrae probability map from step 2. We validated our method on segmentation results from 147 patient images. We computed surface distances between segmentation and ground truth meshes and achieved root mean squared distances of RMSDist = 0.90 mm over all cases and vertebrae.

Cochlear implants (CIs) are neuroprosthetic devices that can improve hearing in patients with severe-to-profound hearing loss. Postoperatively, a CI device needs to be programmed by an audiologist to determine parameter settings that lead to the best outcomes. Recently, our group has developed an image-guided cochlear implant programming (IGCIP) system to simplify the traditionally tedious post-programming procedure and improve hearing outcomes. IGCIP requires image processing techniques to analyze the location of inserted electrode arrays (EAs) with respect to the intra-cochlear anatomy (ICA), and robust and accurate segmentation methods for the ICA are a critical step in the process. We have proposed active shape model (ASM)-based method and deep learning (DL)-based method for this task, and we have observed that DL methods tend to be more accurate than ASM methods while ASM methods tend to be more robust. In this work, we propose a U-Net-like architecture that incorporates ASM segmentation into the network so that it can refine the provided ASM segmentation based on the CT intensity image. Results we have obtained show that the proposed method can achieve the same segmentation accuracy as that of the DL-based method and the same robustness as that of the ASM-based method.

Chest radiographs are a common diagnostic tool in pediatric care, and several computer-augmented decision tasks for radiographs would benefit from knowledge of the anatomic locations within the thorax. For example, a pre-segmented chest radiograph could provide context for algorithms designed for automatic grading of catheters and tubes. This work develops a deep learning approach to automatically segment chest radiographs into multiple regions to provide anatomic context for future automatic methods. This type of segmentation offers challenging aspects in its goal of multi-class segmentation with extreme class imbalance between regions. In an IRB-approved study, pediatric chest radiographs were collected and annotated with custom software in which users drew boundaries around seven regions of the chest: left and right lung, left and right subdiaphragm, spine, mediastinum, and carina. We trained a U-Net-style architecture on 328 annotated radiographs, comparing model performance with various combinations of loss functions, weighting schemes, and data augmentation. On a test set of 70 radiographs, our best-performing model achieved 93.8% mean pixel accuracy and a mean Dice coefficient of 0.83. We find that (1) cross-entropy consistently outperforms generalized Dice loss, (2) light augmentation, including random rotations, improves overall performance, and (3) pre-computed pixel weights that account for class frequency provide small performance boosts. Overall, our approach produces realistic eight-class chest segmentations that can provide anatomic context for line placement and potentially other medical applications.

Convolutional neural networks (CNNs) have been widely and successfully used for medical image segmentation. However, CNNs are typically considered to require large numbers of dedicated expert-segmented training volumes, which may be limiting in practice. This work investigates whether clinically obtained segmentations which are readily available in picture archiving and communication systems (PACS) could provide a possible source of data to train a CNN for segmentation of organs-at-risk (OARs) in radiotherapy treatment planning. In such data, delineations of structures deemed irrelevant to the target clinical use may be lacking. To overcome this issue, we use multi-label instead of multi-class segmentation. We empirically assess how many clinical delineations would be sufficient to train a CNN for the segmentation of OARs and find that increasing the training set size beyond a limited number of images leads to sharply diminishing returns. Moreover, we find that by using multi-label segmentation, missing structures in the reference standard do not have a negative effect on overall segmentation accuracy. These results indicate that segmentations obtained in a clinical workflow can be used to train an accurate OAR segmentation model.

The performance and evaluation of segmentation algorithms will benefit from large fully annotated data sets, but the heavy workload of manual contouring is unrealistic in clinical and research practice. In this work, we propose a method of automatically creating pseudo ground truth (p-GT) segmentations of anatomical objects from given sparse manually annotated slices and utilize them to evaluate actual segmentations. Sparse slices are selected spatially evenly on the whole slice range of the target object, where one slice is selected to conduct manual annotation and the next t slices are skipped, repeating this process starting from one end of the object to its other end. A shape-based interpolation (SI) strategy and an object-specific 2D U-net based deep learning (DL) strategy are investigated to create p-GT. The largest t value where the created p-GT is considered to be not statistically significantly different from the actual ground with its natural imprecision due to variability in manually specified ground truth is determined as the optimal t for the considered object. Experiments are conducted on ~300 computed tomography (CT) studies involving two objects – cervical esophagus and mandible and two segmentation evaluation metrics – Dice Coefficient and average symmetric boundary distance. Results show that the DL strategy overwhelmingly outperforms the SI strategy, where ~95% and ~66-83% of manual workload can be reduced without sacrificing evaluation accuracy compared to actual ground truth data via the DL and SI strategies respectively. Furthermore, the p-GT with optimal t is able to evaluate actual segmentations with accurate metric values.

Medical advancements have been improving the quality of life, but are experiencing an exponential increase in information. Navigating this vast knowledge base is becoming increasingly difficult for doctors without the help of diagnostic assistance tools. Many theorize that advances in deep learning may be poised to transform the field of dermatology, either by helping a general practitioner make a simple dermatological diagnosis or by helping an experienced dermatologist better understand rare or difficult to diagnose skin conditions. While deep learning technology has shown remarkable success in many fields, it still frequently suffers from difficult to explain failures on what appear to be simple input stimuli. Further, these models are forced to make a categorical classification prediction even when an input sample is random noise. We introduce the concept of a "don't know" prediction for dermatological assistance. We firstly modify deep models by replacing neural network weights in the penultimate layer with both random variables and dropout during inference. Statistical ambiguity and entropy features are extracted from a skin lesion classifier and fed into an uncertainty detection model for final prediction estimates. We then present an alternative approach to use machine learning to learn the uncertainty features. Our methods utilize machine learned features without traditional hard-thresholding approaches while minimizing false-positives from skin-lesion inputs on our own skin-lesion dataset as well as the popular SD-198 skin-disease dataset.

Human in-the-loop quality assurance (QA) is typically performed after medical image segmentation to ensure that the systems are performing as intended, as well as identifying and excluding outliers. By performing QA on large-scale, previously unlabeled testing data, categorical QA scores (e.g. “successful” versus “unsuccessful”) can be generated. Unfortunately, the precious use of resources for human in-the-loop QA scores are not typically reused in medical image machine learning, especially to train a deep neural network for image segmentation. Herein, we perform a pilot study to investigate if the QA labels can be used as supplementary supervision to augment the training process in a semi-supervised fashion. In this paper, we propose a semi-supervised multi-organ segmentation deep neural network consisting of a traditional segmentation model generator and a QA involved discriminator. An existing 3-D abdominal segmentation network is employed, while the pre-trained ResNet-18 network is used as discriminator. A large-scale dataset of 2027 volumes are used to train the generator, whose 2-D montage images and segmentation mask with QA scores are used to train the discriminator. To generate the QA scores, the 2-D montage images were reviewed manually and coded 0 (success), 1 (errors consistent with published performance), and 2 (gross failure). Then, the ResNet-18 network was trained with 1623 montage images in equal distribution of all three code labels and achieved an accuracy 94% for classification predictions with 404 montage images withheld for the test cohort. To assess the performance of using the QA supervision, the discriminator was used as a loss function in a multi-organ segmentation pipeline. The inclusion of QA-loss function boosted performance on the unlabeled test dataset from 714 patients to 951 patients over the baseline model. Additionally, the number of failures decreased from 606 (29.90%) to 402 (19.83%). The contributions of the proposed method are threefold: We show that (1) the QA scores can be used as a loss function to perform semi-supervised learning for unlabeled data, (2) the well trained discriminator is learnt by QA score rather than traditional “true/false”, and (3) the performance of multi-organ segmentation on unlabeled datasets can be fine-tuned with more robust and higher accuracy than the original baseline method. The use of QA-inspired loss functions represents a promising area of future research and may permit tighter integration of supervised and semi-supervised learning.

Deep learning methods have shown a high performance potential for medical image analysis [1], particularly classification for computer-aided diagnosis. However, explaining their decisions is not trivial and could be helpful to achieve better results and know how far they can be trusted. Many methods have been developed in order to explain the decisions of classifiers [2]–[7], but their outputs are not always meaningful and remain difficult to interpret. In this paper, we adapted the method of [8] to 3D medical images to find on which basis a network classifies quantitative data. Indeed, quantitative data can be obtained from different medical imaging modalities, for example binding potential maps obtained with positron emission tomography (PET) or gray matter (GM) probability maps extracted from structural magnetic resonance imaging (MRI). Our application focuses on the detection of Alzheimer’s disease (AD), a neurodegenerative syndrome that induces GM atrophy. We used as inputs GM probability maps, a proxy for atrophy, extracted from T1-weighted (T1w) MRI. The process includes two distinct parts: first a convolutional neural network (CNN) is trained to classify AD from control subjects, then the weights of the network are fixed and a mask is trained to prevent the network from classifying correctly all the subjects it has correctly classified after training. The goals of this work are to assess whether the visualization method initially developed for natural images is suitable for 3D medical images and to exploit it to better understand the decisions taken by classification networks. This work is an original work and has not been submitted elsewhere.

Including uncertainty information in the assessment of a segmentation of pathologic structures on medical images, offers the potential to increase trust into deep learning algorithms for the analysis of medical imaging. Here, we examine options to extract uncertainty information from deep learning segmentation models and the influence of the choice of cost functions on these uncertainty measures. To this end we train conventional UNets without dropout, deep UNet ensembles, and Monte-Carlo (MC) dropout UNets to segment lung nodules on low dose CT using either soft Dice or weighted categorical cross-entropy (wcc) as loss functions. We extract voxel-wise uncertainty information from UNet models based on softmax maximum probability and from deep ensembles and MC dropout UNets using mean voxel-wise entropy. Upon visual assessment, areas of high uncertainty are localized in the periphery of segmentations and are in good agreement with incorrectly labelled voxels. Furthermore, we evaluate how well uncertainty measures correlate with segmentation quality (Dice score). Mean uncertainty over the segmented region (Ulabelled) derived from conventional UNet models does not show a strong quantitative relationship with the Dice score (Spearman correlation coefficient of -0.45 for the soft Dice vs -0.64 for the wcc model respectively). By comparison, image-level uncertainty measures derived from soft Dice as well as wcc MC UNet and deep UNet ensemble models correlate well with the Dice score. In conclusion, using uncertainty information offers ways to assess segmentation quality fully automatically without access to ground truth. Models trained using weighted categorical cross-entropy offer more meaningful uncertainty information on a voxel-level.

The quality of chest radiographs is a practical issue because deviations from quality standards cost radiologists' time, may lead to misdiagnosis and hold legal risks. Automatic and reproducible assessment of the most important quality figures on every acquisition can enable a radiology department to measure, maintain, and improve quality rates on an everyday basis. A method is proposed here to automatically quantify the quality according to the aspects of (i) collimation, (ii) patient rotation, and (iii) inhalation state of a chest PA radiograph by localizing a number of anatomical features and calculating some quality figures in accordance with international standards. The anatomical features related to these quality aspects are robustly detected by a combination of three convolutional neural networks and two probabilistic anatomical atlases. An error analysis demonstrates the accuracy and robustness of the method. The implementation proposed here works in real time (less than a second) on a CPU without any GPU support.

Accurate MR-to-CT synthesis is a requirement for MR-only work flows in radiotherapy (RT) treatment planning. In recent years, deep learning-based approaches have shown impressive results in this field. However, to prevent downstream errors in RT treatment planning, it is important that deep learning models are only applied to data for which they are trained and that generated synthetic CT (sCT) images do not contain severe errors. For this, a mechanism for online quality control should be in place. In this work, we use an ensemble of sCT generators and assess their disagreement as a measure of uncertainty of the results. We show that this uncertainty measure can be used for two kinds of online quality control. First, to detect input images that are outside the expected distribution of MR images. Second, to identify sCT images that were generated from suitable MR images but potentially contain errors. Such automatic online quality control for sCT generation is likely to become an integral part of MR-only RT work flows.

We aimed to develop a homology-based approach for prognostic prediction of lung cancer using novel topologically invariant radiomic features. The feasibility of homology-based radiomic features (HFs) was investigated by comparing them with conventional wavelet-based features (WFs) using a Kaplan–Meier analysis for a training dataset (n=135) and a validation dataset (n=70). A total of 13,825 HFs were obtained from histogram and texture features within gross tumor volumes on the computed tomography images using Betti numbers in homology. Similarly, 216 WFs were derived from four wavelet-decomposed images. The prognostic potentials of the HFs were evaluated using statistically significant differences (p-values < 0.05, log-rank test) to compare two survival curves of high- and low-risk patients, which were stratified with medians of radiomic scores of signatures constructed by using an elastic-net-regularized Cox proportional hazard model derived from a Cox-net algorithm. For the training dataset, p-values with hazard ratios (HRs) between the two survival curves were 6.7 × 10^-6 for the HF (HR: 0.41, 95% confidence interval (CI): 0.26-0.65) and 5.9 × 10^-3 for the WF (HR: 0.57, 95%CI: 0.37-0.88). For the validation dataset, p-values with HRs were 3.4 × 10^-5 for the HF (HR: 0.32, 95%CI: 0.16-0.62) and 6.7 × 10^-1 for the WF (HR: 0.88, 95%CI: 0.48-1.6). The HFs showed the more promising potential than the conventional features for prognostic prediction in lung cancer patients.

Accurate lung tumor delineation plays an important role in radiotherapy treatment planning. Since the lung tumor has poor boundary in positron emission tomography (PET) images, it is a challenging task to accurately segment lung tumor. In addition, the heart, liver, bones and other tissues generally have the similar gray value as the lung tumor, therefore the segmentation results usually have high false positive. In this paper, we propose a novel and efficient fully convolutional network with a trainable compressed sensing module and deep supervision mechanism with sparse constraints to comprehensively address these challenges; and we call it fully convolutional network with sparse feature-maps composition (SFC-FCN). Our SFC-FCN is able to conduct end-to-end learning and inference, compress redundant features within channels and extract key uncorrelated features. In addition, we use deep a supervision mechanism with sparse constraints to guide the features extraction by a compressed sensing module. The mechanism is developed by driving an objective function that directly guides the training of both lower and upper layers in the network. We have achieved more accurate segmentation results than that of state-of-the-art approaches with a much faster speed and much fewer parameters.

Positron emission tomography (PET) is a widely used molecular imaging modality for various clinical applications. With Magnetic Resonance Imaging (MRI) providing anatomical information, simultaneous PET/MR reduces the radiation risk. Both improved hardware and algorithms have been developed to further reduce the amount of radiotracer dosage, but these methods are not yet applied to very low dose. Here, we propose a Deep Learning based method to enable ultra-low-dose PET denoising with multi-contrast information from simultaneous MRI. Methods:The method is implemented to denoise 18F-fluorodeoxyglucose (FDG) brain PET images from low-dose images with 200-fold dose reduction through undersampling, and evaluated for glioblastoma (GBM) patients. Comprehensive quantitative and qualitative evaluations were conducted to verify the performance and clinical applicability of the proposed method, including quantitative accuracy evaluation, visual quality evaluation, reader study with manual tumor segmentation to evaluate the diagnostic quality. Results:The results demonstrate that the proposed method achieves superior results in performance and efficiency comparing with the state-of-art denoising methods. Conclusion:Though reconstructed from scans with only 0.5% of the standard dose, the denoised ultra-low-dose PET images deliver similar visual quality and diagnostic information as the standard-dose PET images. By combining PET and MR information, the proposed Deep Learning based method improves image quality of ultra-low-dose PET, preserves diagnostic quality, and potentially enables much safer, faster, and more cost-effective PET/MR studies.

Fluorescence molecular tomography (FMT) is a promising multimodality-fused medical imaging technique, aiming at noninvasively and dynamically visualizing the interaction processes at the cellular and molecular level. However, due to the intricate correlation among columns in system matrix, the inverse problem of FMT faces high ill-posedness. In this work, we propose a novel regularized doubly orthogonal matching pursuit (RDOMP) method through synergistically integrating Gram-Schmidt (GS) orthogonalization with regularized orthogonal matching pursuit (ROMP) to decorrelate the elements in support set against remaining elements. Experiments based on the numerical mouse with double tumors and in vivo mouse were conducted to validate the enhancement of RDOMP with comparison to orthogonal matching pursuit (OMP), sparsity adaptive subspace pursuit (SASP), and ROMP contrast methods. The simulated experimental results demonstrated a better performance of RDOMP in terms of location accuracy, fluorescent yield reconstruction, relative sparsity, and morphological similarity, which highlighted the pruning capability of the introduced GS orthogonalization. And in vivo experimental results demonstrated the practical application of RDOMP on FMT reconstruction.

PET/CT is widely used in oncology. Yet the identification of lesions, as described by the PET response criteria in solid tumors (PERCIST), still relies on manual identification of a volume of interest (VOI), typically in the liver, for determining the optimal threshold. The process requires expert knowledge and is prone to errors and inter-observer variability. A fully automated procedure for the application of the PERCIST criteria for whole- body images is proposed. The method relies on automated localization of the liver on whole-body CT using a dense V-net trained on large field-of-view images. Inside the liver, a spherical VOI is determined which exhibits the lowest product of the coefficients of variation (defined as the standard deviation over the mean) in PET and CT. The liver segmentation achieved a median dice score of 0.87 ± 0.12 in 10-fold cross-validation, which proved to be sufficient for reliable identification of a VOI. The full pipeline was evaluated on an external PET/CT dataset of 18 patients. To assess reproducibility, geometric and intensity variations were applied, simulating potential image differences when scanning the same person under different conditions. The variability of the resulting threshold was evaluated and compared to the manual approach performed by three observers. The proposed method exhibited superior reproducibility with a mean threshold of 4.01 ± 0.02 SUV_bw, compared to 4.11 ± 0.16 SUV_bw for the manual method. The automated procedure renders the analysis of large amounts of PET/CT data feasible or could be used to detect anomalies in the manual approach.

The tongue is capable of producing intelligible speech because of successful orchestration of muscle groupings— i.e., functional units—of the highly complex muscles over time. Due to the different motions that tongues produce, functional units are transitional structures which transform muscle activity to surface tongue geometry and they vary significantly from one subject to another. In order to compare and contrast the location and size of functional units in the presence of such substantial inter-person variability, it is essential to study both common and subject-specific functional units in a group of people carrying out the same speech task. In this work, a new normalization technique is presented to simultaneously identify the common and subject-specific functional units defined in the tongue when tracked by tagged magnetic resonance imaging. To achieve our goal, a joint sparse non-negative matrix factorization framework is used, which learns a set of building blocks and subject-specific as well as common weighting matrices from motion quantities extracted from displacements. A spectral clustering technique is then applied to the subject-specific and common weighting matrices to determine the subject-specific functional units for each subject and the common functional units across subjects. Our experimental results using in vivo tongue motion data show that our approach is able to identify the common and subject-specific functional units with reduced size variability of tongue motion during speech.

The identification of mental load has important practical significance for human life. In recent years, studies have used physiological signals of different modalities to measure mental load. However, there is no single indicator that can fully assess mental load, and the fusion of multimodal signals to measure this variable is still controversial. The emergence of simultaneous EEG and fMRI enables researchers to explore brain function, especially working memory, with high temporal and spatial resolution. In this study, we fused EEG and fMRI data on network features to estimate working memory load. By introducing a filter bank, the phase synchronization values covering the full frequency band were extracted as the EEG network features. Then, we proposed the Wasserstein distance to measure functional connectivity in single-trial fMRI data. Finally, Fisher vector was used to fuse these features from the above two modalities, and this method was compared with the direct splicing features. The results showed that the Fisher vector was more effective for recognition than direct splicing was, providing a new option for multimodality feature fusion. In addition, distance-based functional connectivity extraction on the individual level enriches the range of tools with which to study brain network.

The study of brain network connectivity as a time‐varying property began relatively recently and to date has remained primarily concerned with capturing a handful of discrete static states that characterize connectivity as measured on a timescale shorter than that of the full scan. Capturing representations of temporally evolving patterns of connectivity is a challenging and important next step in fully leveraging the information available in fMRI data. We introduce a constellation of interrelated data‐driven methods that hierarchically employ multichannel 1D sparse convolutional dictionary learning (SCDL) and independent component analysis (ICA) for extracting multiscale time‐varying representations of functional network connectivity (FNC). This work also relies upon a novel wavelet‐based method for computing dynamically varying FNC (dFNC) at each timepoint in the scan, yielding a much more resolved picture of evolving connectivity than currently popular sliding‐window approaches. The methods are validated in application to a large multisite fMRI study of schizophrenia where they expose properties of time‐varying connectivity in schizophrenia patients vs. controls that are surprising based on long‐accepted theories of the disorder.

Segmentation is a critical step in medical image analysis. Fully Convolutional Networks (FCNs) have emerged as powerful segmentation models achieving state-of-the art results in various medical image datasets. Network architectures are usually designed manually for a specific segmentation task so applying them to other medical datasets requires extensive expertise and time. Moreover, the segmentation requires handling large volumetric data that results in big and complex architectures. Recently, methods that automatically design neural networks for medical image segmentation have been presented; however, most approaches either do not fully consider volumetric information or do not optimize the size of the network. In this paper, we propose a novel self-adaptive 2D-3D ensemble of FCNs for 3D medical image segmentation that incorporates volumetric information and optimizes both the models performance and size. The model is composed of an ensemble of a 2D FCN that extracts intra-slice and long-range information, and a 3D FCN that exploits inter-slice information. The architectures of the 2D and 3D FCNs are automatically adapted to a medical image dataset using a multiobjective evolutionary based algorithm that minimizes both the expected segmentation error and number of parameters in the network. The proposed 2D-3D FCN ensemble was tested on the task of prostate segmentation on the image dataset from the PROMISE12 Grand Challenge. The resulting network is ranked in the top 10 submissions, surpassing the performance of other automatically-designed architectures while having 13.3x fewer parameters.

The prevalence of myopia is rapidly increasing worldwide. Along with the deepening of myopia, there will be various pathological changes of retina, such as choroidal atrophy, choroidal neovascularization, etc. In this paper, a U-Net based deep network is proposed to automatically segment choroidal atrophy in fundus images. We use U-Net as the main structure, which can learn rich hierarchical feature representations. In the decoder path, Squeeze-and-Excitation (SE) block is employed before each deconvolution to adaptively recalibrate channel feature response. We introduce deep-supervision mechanism and merge all the early prediction maps to obtain final prediction map. To ensure the smoothness of segmentation results, we propose a new loss function, which is termed EDT-auxiliary-loss (Euclidean distance transformation auxiliary loss). EDT-auxiliary-loss consists of Dice loss for ground truth and mean square error (MSE) loss for distance map. Another strategy for performance improvement is utilizing the information of optic disc (OD), which is usually adjacent to atrophy. The proposed method was evaluated on ISBI 2019 Pathologic Myopia Challenge dataset, which consists of 400 fundus images from161 normal eyes, 26 high myopia eyes and 213 pathologic myopia eyes. The proposed network was validated with four-fold cross validation. The experiment results show that the proposed method can successfully segment choroidal atrophy and achieve better performance than traditional U-Net.

Whole heart segmentation from cardiac CT scans is a prerequisite for many clinical applications, but manual delineation is a tedious task and subject to both intra- and inter-observer variation. Automating the segmentation process has thus become an increasingly popular task in the field of image analysis, and is generally solved by either using 3D methods, considering the image volume as a whole, or 2D methods, segmenting each slice independently. In the field of deep learning, there are significant limitations regarding 3D networks, including the need for more training examples and GPU memory. The need for GPU memory is usually solved by down sampling the input images, thus losing important information, which is not a necessary sacrifice when employing 2D networks. It would therefore be relevant to exploit the benefits of 2D networks in a configuration, where spatial information across slices is kept, as when employing 3D networks. The proposed method performs multiclass segmentation of cardiac CT scans utilizing 2D convolutional neural networks with a multi-planar approach. Furthermore, spatial propagation is included in the network structure, to ensure spatial consistency through each image volume. The approach keeps the computational assets of 2D methods while addressing 3D issues regarding spatial context. The pipeline is structured in a two-step approach, in which the first step detects the location of the heart and crops a region of interest, and the second step performs multi-class segmentation of the heart structures. The pipeline demonstrated promising results on the MICCAI 2017 Multi-Modality Whole Heart Segmentation challenge data.

Corneal confocal microscopy (CCM) is a new technique offering non-invasive and fast imaging useful for diagnosing and analyzing corneal diseases. The morphology of corneal nerve fibres can be clearly observed from CCM images. Segmentation and quantification of nerve fibres is important for analyzing corneal diseases such as diabetic peripheral neuropathy (DPN). In this paper, we propose an automated deep learning based method for corneal nerve fibre segmentation in CCM images. The main contributions of this paper are: (1)We add multi-scale split and concatenate (MSC) blocks to the decoding part of the four layer U-Net architecture. (2) A new loss function is applied that combining the Dice loss with the fibre length difference between the ground truth and the prediction. The method was tested on a dataset containing 90 CCM images from 4 normal eyes and 4 eyes with corneal diseases. The Dice coefficient of our approach can reach 87.96%, improves 1.6% compared with the baseline, and outperforms some existing deep networks for segmentation.

Choroidal neovascularization(CNV) refers to abnormal choroidal vessels that grow through the Bruch’s membrane to the bottom of retinal pigment epithelium (RPE) or retinal neurepithelium (RNE) layer, which is the pathological characterization of age-related macular degeneration (AMD) and pathological myopia (PM). Nowadays, optical coherence tomography (OCT) is an important imaging modality for observing CNV. This paper creatively proposes a convolutional neural network with differential amplification blocks (DACNN) to segment CNV in OCT images. There are two main contributions. (1) A differential amplification block (DAB) is proposed to extract the contrast information of foreground and background. (2) The DAB is integrated into the U-shaped convolutional neural network for CNV segmentation. The method proposed in this paper was verified on a dataset composed of 886 OCT B-scans. Compared with manual segmentation, the mean Dice similarity coefficient can reach 86.40%, outperforming some existing deep networks for segmentation.

Retinopathy of prematurity (ROP) is an ocular disease which occurs in premature babies and is considered as one of the largest preventable causes of childhood blindness. However, insufficient ophthalmologists are qualified for ROP screening, especially in developing countries. Therefore, automated screening of ROP is particularly important. In this paper, we propose a new ROP screening network, in which pre-trained ResNet18 is taken as backbone and a proposed attention block named Complementary Residual Attention Block (CRAB) and Squeeze-and-Excitation (SE) block as channel attention module are introduced. Our main contributions are: (1) Demonstrating the 2D convolutional neural network model pre-trained on natural images can be fine-tuned for ROP screening. (2) Based on the pre-trained ResNet18, we propose an improved scheme combining which that effectively integrates attention mechanism for ROP screening. The proposed classification network was evaluated on 9794 fundus images from 650 subjects, in which 8351 are randomly selected as training set according to subjects and others are selected as testing set. The results showed that the performance of the proposed ROP screening network achieved 99.17% for accuracy, 98.65% for precision, 98.31% for recall, 98.48% for F1 score and 99.84% for AUC. The preliminary experimental results show the effectiveness of the proposed method.

Oncology PET protocols utilize administered activities ranging between 10-15 mCi and 2-3 min/bed positions to obtain diagnostic quality PET images. There is a continued desire to reduce administered activity to reduce radiation exposure and to decrease scanning time to improve patient comfort. However, reducing those parameters lowers photon counts and degrades quantification accuracy. In this work, we proposed a deep-learning-based method to estimate the diagnostic PET image from low count data. A cycle-consistent generative adversarial network (Cycle GAN) was introduced to capture the relationship from low count to full count PET images while simultaneously supervising an inverse full count to low count (full-to-low) transformation model. The network simultaneously makes itself better at both creating synthetic full count PET images and learns how to identify full count PET images. Residual blocks were integrated into the models to catch the differences between low count and full count PET in the training dataset and better handle noise. The proposed model was implemented and evaluated on whole-body FDG PET images with only 1/8th counts. The proposed method obtained the average mean error and normalized mean square error in the whole body of 0.14%±1.43% and 0.52%±0.19%. Normalized cross-correlation was increased to 0.996, and the peak signal-to-noise ratio is increased to 46.0 dB with the proposed method. We developed a deep learning-based approach to accurately estimate diagnostic quality PET datasets from one-eighth of photons, with great potential to substantially reduce the administered dose or scan duration while maintaining high diagnostic quality.

Deep learning has achieved many successes in medical imaging, including lung nodule segmentation and lung cancer prediction on computed tomography (CT). Recently, multi-task networks have shown to both offer additional estimation capabilities, and, perhaps more importantly, increased performance over single-task networks on a "main/primary" task. However, balancing the optimization criteria of multi-task networks across different tasks is an area of active exploration. Here, we extend a previously proposed 3D attention-based network with four additional multi-task subnetworks for the detection of lung cancer and four auxiliary tasks (diagnosis of asthma, chronic bronchitis, chronic obstructive pulmonary disease, and emphysema). We introduce and evaluate a learning policy, Periodic Focusing Learning Policy (PFLP), that alternates the dominance of tasks throughout the training. To improve performance on the primary task, we propose an Internal-Transfer Weighting (ITW) strategy to suppress the loss functions on auxiliary tasks for the final stages of training. To evaluate this approach, we examined 3386 patients (single scan per patient) from the National Lung Screening Trial (NLST) and de-identified data from the Vanderbilt Lung Screening Program, with a 2517/277/592 (scans) split for training, validation, and testing. Baseline networks include a single-task strategy and a multi-task strategy without adaptive weights (PFLP/ITW), while primary experiments are multi-task trials with either PFLP or ITW or both. On the test set for lung cancer prediction, the baseline single-task network achieved prediction AUC of 0.8080 and multi-task baseline failed to converge (AUC 0.6720). However, applying PFLP helped multi-task network clarify and achieved test set lung cancer prediction AUC of 0.8402. Furthermore, our ITW technique boosted the PFLP enabled multi-task network and achieved an AUC of 0.8462 (McNemar test, p < 0.01). In conclusion, adaptive consideration of multi-task learning weights is important, and PFLP and ITW are promising strategies.

Patient motion during computed tomography (CT) scan can result in serious degradation of imaging quality, and is of increasing concern due to the aging population and associated diseases. In this paper, we address this problem by focusing on the reduction of head motion artifacts. To achieve this, we introduce a head motion simulation system and a multi-scale deep learning architecture. The proposed motion simulation system can simulate rigid movement including translation and rotation. The images with simulated motion serve as the training set for the network, and the original motion free images serve as the gold standard. Motion artifacts exhibit in the image space as streaks and patchy shadows. We propose a multiscale neural network to learn the artifact. With different branches equipped with ResBlock and down-sampling, the network can learn long scale streaks and short scale shadow artifacts. Although we trained the network on simulated images, we find that the learned network generalizes well to images with real motion artifacts.

An automatic segmentation of liver lesion is one of the essential processes for computer-aided diagnosis for screening of liver diseases. In this study, we proposed an in-house, reinforced U-Net, i.e., the ‘CT attenuation-integrated U-Net (CAIUNet)’ as a new deep learning model for automatic segmentation of focal liver lesion in abdominal CT imaging. The CAIUNet is based on the basic U-Net. The CAI-UNet is focused on the CT attenuation value, which is significant information to differentiate between healthy tissues and lesions in CT imaging, but could not be directly included after passing through several convolutional operations in the basic U-Net. We introduced directly the connection between outputs of the last 3x3 convolution and a raw input image to enhance CT attenuation information. For training CAI-UNet, the weighted dice loss function was used to solve the imbalance of target lesions. For evaluation, we used LiTS challenge dataset of 131 abdominal CT which contained various focal liver lesions, and selected 90 sets containing liver metastasis by a radiologist with more than 30 years of experience. For statistical analysis, we performed a paired t-test to compare the lesion segmentation accuracy. Our results showed that CAI-UNet in performing liver lesion segmentation yielded 0.646 global dice score, 0.543 subject dice score, 0.568 specificity score and 0.651 precision score on test dataset. We also found that the CAI-UNet showed the significant improvement for segmentation of liver lesions compared with U-Net only (P < 0.05). The proposed CAI-UNet could be used to improve the detection of liver lesion.

Ischemic stroke infarct tissues are not salvageable. The infarct volume calculated from a segmented infarct region is an important parameter required to decide on the optimal treatment work ow. Deep learning continues to demonstrate the significance of end-to-end training with limited use of apriori knowledge (such as domain-aware feature engineering) in learning medical imaging tasks. Incorporating prior domain-specific knowledge introduces better inductive bias in learning tasks with low data availability, thereby improving performance. Several techniques have been used for segmentation of infarct region ranging from traditional approaches like region growing to deep learning approaches with limited use of domain-specific knowledge. This paper incorporates domain-specific knowledge into deep neural networks to restrict the region of interest thereby improving the performance of infarct segmentation. Incorporating domain-specific knowledge improve the performance by 17%.

Anti-scatter grids are routinely used to prevent degradation of image quality caused by scattered X-ray beams. However, these grids might cause linear artifacts that represent the shadows of the radiopaque septa. In this paper, we propose a machine learning-based method for grid artifact suppression in radiography. There are two major difficulties in the application of a deep learning technique for the grid-line suppression problem. The first is quantitative shortage of learning data. It is difficult to acquire a sufficient amount of learning data from observed images in consideration of the various situations in which grid lines appear. The second is difficulty determining the target data. It is practically impossible to generate a clean target image for an arbitrary grid-line image. To overcome these problems, we propose a deep convolutional neural network architecture and a learning data construction method. A discrete cosine transform-based band-stop filtering technique and an image synthesizing algorithm were adopted for the learning data construction method. A patch sampling method was employed to overcome the shortage of the amount of learning data. The proposed method enables learning without clean target data and overcomes the weakness of conventional frequency analysis-based methods with regard to the grid-line suppression problem. This method makes it possible to expect complementary performance through the construction of a combined learning data set including observed images and artificially generated grid-line images.

Anatomical landmark correspondences in medical images can provide additional guidance information for the alignment of two images, which, in turn, is crucial for many medical applications. However, manual landmark annotation is labor-intensive. Therefore, we propose an end-to-end deep learning approach to automatically detect landmark correspondences in pairs of two-dimensional (2D) images. Our approach consists of a Siamese neural network, which is trained to identify salient locations in images as landmarks and predict matching probabilities for landmark pairs from two different images. We trained our approach on 2D transverse slices from 168 lower abdominal Computed Tomography (CT) scans. We tested the approach on 22,206 pairs of 2D slices with varying levels of intensity, affine, and elastic transformations. The proposed approach finds an average of 639, 466, and 370 landmark matches per image pair for intensity, affine, and elastic transformations, respectively, with spatial matching errors of at most 1 mm. Further, more than 99% of the landmark pairs are within a spatial matching error of 2 mm, 4 mm, and 8 mm for image pairs with intensity, affine, and elastic transformations, respectively. To investigate the utility of our developed approach in a clinical setting, we also tested our approach on pairs of transverse slices selected from follow-up CT scans of three patients. Visual inspection of the results revealed landmark matches in both bony anatomical regions as well as in soft tissues lacking prominent intensity gradients.

In this study, we propose to use an unsupervised deep learning-based method to directly register the MRI to CT with the help of synthetic MRI (sMRI). Our synthesis results showed that CT-generated sMRI could partially restore soft-tissue contrast in the pelvic region, which helps to improve the image registration accuracy near the prostate, rectum and bladder. Additionally, sMRI has similar intensity value to the true MRI which make it easier to register the images. The registration network was trained in an unsupervised manner, meaning ground truth DVF is not required. The trained network can predict the deformation vector field (DVF) in a single shot. After registration, the structural similarity index metric (SSIM) and peak signal to noise ratio (PSNR) between the deformed sMRI and the fixed MRI were on average 0.79 and 25.15 dB respectively.

Image quality assessment (IQA) of carotid vessel walls from magnetic resonance imaging (MRI) is critical to accurate diagnosis and prevention of stroke. However, most existing solutions for IQA are either manual or based only on holistic information. The low efficiency and accuracy of these methods hampers the transition of vessel wall imaging into clinical use. In this paper, we propose an IQA framework which assesses image quality using local features from multiple patches close to the target region in the image. Following criterion for target-oriented medical imaging quality assessment, we highlight the patch covering the artery detected by a neural network built on YOLOv2 and set the weights for other patches based on the human visual system both in training and testing. Finally, the image score is determined by a weighted average of patch scores. This method proved able to identify and quantify image quality using MRI datasets of different sequences with over 82% sensitivity and 90% specificity for four sequences (3D-MERGE, T1, T2, TOF) separately tasked with binary classification. Our proposed system shows the method’s advantages on accuracy, efficiency, and adaptability in clinical use.

Throughout human lifetime there are various ageing-related changes occurring. It has been demonstrated that departures from the healthy ageing trajectory can be used as a biomarker for several neurodegenerative conditions. Previous neuroimaging studies have used deep learning algorithms to investigate structural biomarkers including volumetric, microstructural and focal ones, using both cross-sectional and longitudinal cohorts. In this work, we would like to compare the metric and ordinal regression on the task of brain age prediction and devise a healthy whole brain ageing profile. For this purpose we have assembled a dataset comprising of 10,878 MRI T1-weighted scans acquired in healthy subjects. We have used the dataset to train a convolutional neural network-based ordinal regression model for age prediction using the imaging data with minimal pre-preprocessing (affine transformation and resampling onto a template). We leverage the distribution of predicted ages to propose ageing profile by assuming that the distribution of age predictions represents the distribution of age-related structural features. For comparison brain-ageing profile is obtained using the method of deep embedded clustering (DEC). Our ordinal regression model achieved a mean absolute error of 3.6 years. The results show that end-to-end learning of imaging features using an ordinal regression loss can yield age predictions that are comparable to, or even better, than metric-based regression models. Using DEC and the distribution of predictions resulted in an ageing profile consisting of 10 to 12 intervals. Resulting intervals in the ageing profile highlight the nonlinear nature of the ageing process in agreement with existing literature.

We evaluate the use of TernausNet V2, a pre-trained VGG-16 U-net for segmentation of Green Fluorescent Protein (GFP) stained stem cells from giga pixel fluorescence microscopy images. Fluorescence microscopy is a difficult modality for automated stem cell segmentation algorithms due to high noise and low contrast. As such segmentation algorithms for cell counting and tracking typically yield more consistent results in other imaging modalities such as Phase Contrast (PC) microscopy due to greater ability to distinguish between foreground and background. Recent methods have shown that U-net based models can achieve state-of-the-art segmentation performance of GFP microscopy, although all available methods continue to overly segment the protein features and have difficulty capturing the entirety the cell. We investigate the use of TernausNet, a VGG-16 based U-Net architecture that was pre-trained from ImageNet and show that it is able to improve the accuracy of GFP stem cell segmentation on gigascale NIST fluorescence microscopy images in comparison to a baseline U-net model. Quantitative results show that the proposed TernausNet V2 architecture model is able to better distinguish the entire region of the cell and reduce overly segmenting proteins as compared to U-net. TernausNet achieved greater accuracy with ROC AUC of 0.956 and F1-Score of 0.810 as compared to the baseline U-net with AUC 0.936 and F1-Score 0.775. Therefore, we suggest that the TernausNet V2 architecture with transfer learning improves the performance of stem-cell segmentation is able to outperform U-net models for the segmentation of giga pixel GFP stained fluorescence microscopy images.

Epicardial fat is a visceral fat deposit, located between the heart and the pericardium, which shares many of the pathophysiological properties of other visceral fat deposits, it may also potentially cause local inflammation and likely has direct effects on coronary atherosclerosis. epicardial fat is also associated with other known factors, such as obesity, diabetes mellitus, age, and hypertension, which interprets its role as an independent risk marker intricate. For the investigation of the relationship between epicardial fat and various diseases, it is important to segment the epicardial fat in a fast and reproducible way. However, epicardial fat has a variable distribution, and multiple conditions may affect the volume of the EF, which can increase the complexity of the already time-consuming manual segmentation work. In this study, we propose to use a 3D deep attention U-Net method to segment the epicardial fat for cardiac CT image automatically. To test the proposed method, we applied it to 40 patients’ cardiac CT images. Five-fold cross-validation experiments were used to evaluate the proposed method. We calculated the Dice similarity coefficient (DSC), precision, and recall (MSD) indices between the ground truth and our segmentation to quantify the segmentation accuracy of the proposed method. Overall, the DSC, precision, and recall were 85% ± 5%, 86% ± 4%, and 89% ± 5%, which demonstrated the detection and segmentation accuracy of the proposed method.

Optimization of loss function is one of the research directions in medical image registration. A loss function of registration is the sum of two terms: a similarity term L_sim (Φ) and a smoothing term L_smooth(Φ). From variational method in differential geometry, control function is essential to generate better registration field Φ. Here, we propose a new registration loss function with novel smoothing terms using VoxelMorph based on control function and Laplacian operator. We divide the process into two steps. The first step is based on Laplacian operator. We replace the gradient of registration field Φ in L_smooth (Φ) by the Laplacian of Φ. In the second step, we add the term control function F to the L_smooth (Φ) in the first step, which is the key contribution of our method. We verify our method on two datasets including ADNI and IBSR, and obtain excellent improvement on MR image registration, with better convergence and gets higher average Dice and lower percentage of non-positive Jacobian locations compared with original loss function.

Functional magnetic resonance imaging (fMRI) has been implemented widely to study brain connectivity. In the context of fMRI, independent component analysis (ICA) is a powerful tool, which extracts patterns from the data without requiring prior knowledge. Recently, time-varying connectivity analysis has emerged as an important measure to uncover essential knowledge within the network. In this study, we propose a new framework that combines group ICA (GICA) with time varying graphical LASSO (TVGL) to improve the power of analyzing functional network connectivity (FNC) changes. To investigate the performance of our proposed approach, we apply it to capture dynamic FNC using the Pediatric Imaging, Neurocognition, and Genetics (PING) datasets. Our results indicate that females and males of young adults do not show large FNC differences though some slight variations have been found. For instance, females exhibited stronger interdomain FNC and greater correlation in occipital-frontal components for some specific states in comparison to males. In addition, the TVGL-GICA model indicated that females had a higher probability to stay in a stable state. Males had a higher tendency to remain in a globally disconnected mode. Our proposed framework provides a feasible method to investigate brain dynamics accurately and has the potential to become a useful tool in neuroimaging studies.

Radon transforms allow to represent n-dimensional objects by all their possible (n-1)-dimensional integrals. They find broad usage in a variety of image processing topics, covering pattern recognition and tomographic imaging. Potentially the most frequently used version is the 2D Radon transform which is commonly computed by means of the ray tracing procedure (Siddon, Joseph etc.).¹ The problem comes down to a method of approximating a line integral through a 2D pixelized image. For higher dimensions, however, this problem becomes more and more complex and typically involves a substantial amount of case differentiation, making it particularly ill-suited for use in massively parallelized computation (e.g. on GPUs). Additionally, implementation effort is substantial and quite error-prone. Here, we propose a simple strategy to compute the (n-1)-dimensional integrals in a generalized manner by reducing the sampling problem to a single matrix multiplication. We further present OpenCL implementations for n=2 and n=3, making use of hardware interpolation methods on texture memory of GPU devices to provide a fast computation of the transform.

Suppression of speckle artifact in optical coherence tomography (OCT) is necessary for high quality quantitative assessment of ocular disorders associated with vision loss. However, due to its dual role as a source of noise and as a carrier of information about tissue microstructure, complete suppression of speckle is not desirable. That is what represents challenge in development of methods for speckle suppression. We propose method for additive decomposition of a matrix into low-rank and group sparsity constrained terms. Group sparsity constraint represents novelty in relation to state-of-the-art in low-rank sparse additive matrix decompositions. Group sparsity enforces more noise-related speckle to be absorbed by the sparse term of decomposition. Thus, the low-rank term is expected to enhance the OCT image further. In particular, proposed method uses the elastic net regularizer to induce the grouping effect. Its proximity operator is shrunken version of the soft-thresholding operator. Thus, the group sparsity regularization adds no extra computational complexity in comparison with the ℓ₁ norm regularized problem. We derive alternating direction method of multipliers based algorithm for related optimization problem. New method for speckle suppression is automatic and computationally efficient. The method is validated in comparison with state-of-the-art on ten 3D macular-centered OCT images of normal eyes. It yields OCT image with improved contrast-to-noise ratio, signal-to-noise ratio, contrast and edge fidelity (sharpness).

Metal artifacts are very common in CT scans since many patients have metal insertion or replacement to enhance functionality or mechanism of their bodies. These streaking artifacts could degrade CT image quality severely, and consequently, they could influence clinical diagnosis. In this paper, we propose to use the Fourier coefficients of a metal artifact-tainted image as the input to a convolutional neural network, and the Fourier coefficients of the corresponding clean image as target. We compare the performances of three convolutional neural network models with three kinds of inputs - sinograms with metal traces, images with streaks, and the Fourier coefficients of artifact-corrupted images. Using Fourier coefficients as inputs gives generally better artifacts reduction results in visualization and quantitative measures in different models.

We propose a deep-learning-based method to reconstruct super-resolution images from routinely captured MRI images. We propose to integrate a deeply supervised attention model into a generative adversarial network (GAN)-based framework to improve MRI image resolution. Deep attention GANs are introduced to enable end-to-end encoding-and-decoding learning. Next, an attention model is used to retrieve the most relevant information from the encoder. The residual network is used to learn the difference between low- and high-resolution images. This technique was validated with 20 patients. We performed a leave-one-out cross-validation method to evaluate the proposed algorithm and further tested it with a down-sampling rate 1/3 and 1/6. Our reconstructed high-resolution MRI images from down-sampling images were compared with the original image to evaluate the performance quantitatively. The reconstructed super-resolution images were compared to a high-resolution reference scan using and the mean absolute error (MAE) and peak signal-to-noise ratio (PSNR) of image intensity profiles. The MAE and PSNR between reconstructed and original images for 1/3 down sampling rate were 3.86±1.53 and 41.95 ± 5.06dB, and 9.45 ± 1.73, 35.05 ± 3.64dB for 1/6 respectively, which demonstrates the accuracy of the proposed method.

In many applications based on kinetic evaluation analysis and model fitting, quantitative mapping retrieved on data series from modalites such as MRI is completed on a voxel-by-voxel basis, where motion and low signal to noise ratio (SNR) would considerably degenerate the reliability of estimations. The coherence of image series in space and time can be used as prior knowledge to mitigate this occurrence. In this study, spatial and temporal higher-order total variations (HOTVs) are applied on a data series of MRI signal (e.g. dynamic contrast-enhanced (DCE) MRI and intravoxel incoherent motion (IVIM) MRI) to exploit the coherence of signal in space and time to minimize the variabilities caused by motion as well as improving quality of images with low SNR while retaining the physical details of original data properly. Simultaneously applying spatial and temporal HOTVs on images is non-trivial in implementation since it is a non-smooth optimization problem with multiple regularizers. Therefore, we use the proximal gradient method as well as a primal-dual split proximal mechanism to address the problem properly. In addition to increase the reliability of quantitative parametric map estimations, this preprocessing procedure can be included into many existing map estimation algorithms and pipelines effortlessly. We demonstrate our method on the parametric maps estimation for DCE MRI and IVIM MRI.

The application of MRI significantly improves the accuracy and reliability of target delineation for many disease sites in radiotherapy due to its superior soft tissue contrast as compared to CT. However, MRI data do not contain the electron density information that is necessary for accurate dose calculation. There has been limited work in abdominal synthetic CT (sCT) generation. In this work, we propose to integrate dense blocks and a novel compound loss function into a 3D cycleGAN-based framework to generate sCT from MR images. Since MRI and CT are two different image modalities, dense blocks are employed to combine low- and high-frequency information that can effectively represent different image patches. A novel compound loss function with l_p-norm (p = 1.5) distance and gradient difference is used to differentiate the structure boundaries and to retain the sharpness of the sCT image. This proposed algorithm was evaluated using 21 hepatocellular cancer patients’ registered MR and CT images as the training dataset. Leave-one-out cross-validation was performed. The average mean absolute error (MAE), peak signal-to-noise ratio (PSNR), and normalized cross correlation (NCC) were 72.87±18.16 HU, 22.65±3.63 dB, and 0.92±0.04 respectively. The proposed method has encouraging outcomes in generating sCT for the potential use in MR-only photon or proton radiotherapy treatment planning.

Medical imaging datasets typically do not contain many training images and are usually not sufficient for training deep learning networks. We propose a deep residual variational auto-encoder and a generative adversarial network based approach that can generate a synthetic retinal fundus image dataset with corresponding blood vessel annotations. In terms of structural statistics comparison of real and artificial our model performed better than existing methods. The generated blood vessel structures achieved a structural similarity value of 0.74 and the artificial dataset achieved a sensitivity of 0.84 and specific city of 0.97 for the blood vessel segmentation task. The successful application of generative models for the generation of synthetic medical data will not only help to mitigate the small dataset problem but will also address the privacy concerns associated with such medical datasets.

Similarity measures are a critical component for iterative registration tasks. In intrinsic 2D/3D registration an initialization close to the solution is usually needed for convergence. Traditional similarity measures are limited in their applicability to perspective projection data. Deep Learning allows us to create similarity measures that encode almost arbitrary non-linear relationships like perspective projection. We apply a siamese network and a 2-channel network to the problem of comparing two perspective maximum intensity projections of medical data. We propose the use of residual units as the building block. A major challenge is to define similarity under perspective projection so that it can be used for training. Extrinsic registration gives us a gold standard of registered examples but we need a way to quantify the dissimilarity of unregistered image pairs for training. We propose the use of disparity as the basis for construction of similarity labels, i.e. labels are based on distances between projected points when viewing the same 3D point through different cameras. The final similarity labels are independent of the choice of camera parametrization and describe the visual similarity of two given views of a common 3D model. We compare our deep similarity metric to traditional similarity measures for a comparative evaluation. Our best configuration has a Pearson correlation coefficient of 0.792 and a Spearman’s rank correlation coefficient of 0.480. The best traditional method is normalized cross correlation with a 5.4% lower Spearman’s rank correlation coefficient.

Segmentation of abdominal computed tomography (CT) provides spatial context, morphological properties, and a framework for tissue-specific radiomics to guide quantitative Radiological assessment. A 2015 MICCAI challenge spurred substantial innovation in multi-organ abdominal CT segmentation with both traditional and deep learning methods. Recent innovations in deep methods have driven performance toward levels for which clinical translation is appealing. However, continued cross-validation on open datasets presents the risk of indirect knowledge contamination and could result in circular reasoning. Moreover, “real world” segmentations can be challenging due to the wide variability of abdomen physiology within patients. Herein, we perform two data retrievals to capture clinically acquired deidentified abdominal CT cohorts with respect to a recently published variation on 3D U-Net (baseline algorithm). First, we retrieved 2004 deidentified studies on 476 patients with diagnosis codes involving spleen abnormalities (cohort A). Second, we retrieved 4313 deidentified studies on 1754 patients without diagnosis codes involving spleen abnormalities (cohort B). We perform prospective evaluation of the existing algorithm on both cohorts, yielding 13% and 8% failure rate, respectively. Then, we identified 51 subjects in cohort A with segmentation failures and manually corrected the liver and gallbladder labels. We re-trained the model adding the manual labels, resulting in performance improvement of 9% and 6% failure rate for the A and B cohorts, respectively. In summary, the performance of the baseline on the prospective cohorts was similar to that on previously published datasets. Moreover, adding data from the first cohort substantively improved performance when evaluated on the second withheld validation cohort.

Deep neural networks (DNNs) have been widely used in the medical imaging field. The large and high quality dataset is crucial for the performance of the deep learning models, but the medical data and ground-truth is often insufficient and very expensive in terms of time and human effort on the data collection. However, we can improve the performance of the deep learning model by augmenting the data we already have. In this work, we introduce a novel differential geometry-based quasi conformal (QC) mapping augmentation technique to augment the brain tumor images. The QC method lets the user specify or randomly generate a complex-valued function on the image domain via Beltrami coefficient. By solving the Beltrami equation with given Beltrami coefficient, the QC map, which can further guide the deformation of the image, is able to generate all possible linear and non-linear image warpings and it is flexible to allow the user to fully control the global and local deformations. Our experimental results demonstrate the efficiency and efficacy of the proposed method.

Some veterans with a history of mild traumatic brain injury (mTBI) have reported experiencing auditory and visual dysfunction that persist beyond the acute phase of the incident. The etiology behind these symptoms is difficult to characterize, since mTBI is defined by negative imaging findings on current clinical imaging. There are several competing hypotheses that could explain functional deficits; one example is shear injury, which may manifest in diffusion-weighted magnetic resonance (MR) imaging (DWI). Herein, we explore this alternative hypothesis in a pilot study of multiparametric MR imaging. Briefly, we consider a cohort of 8 mTBI patients relative to 22 control subjects using structural T1-weighted imaging (T1w) and connectivity with DWI. 1,344 metrics were extracted per subject from whole brain regions and connectivity patterns in sensory networks. For each set of imaging-derived metrics, the control subject metrics were embedded in a low-dimensional manifold with principal component analysis, after which mTBI subject metrics were projected into the same space. These manifolds were employed to train support vector machines (SVM) to classify subjects as controls or mTBI. Two of the SVMs trained achieved near-perfect accuracy averaged across four-fold cross-validation. Additionally, we present correlations between manifold dimensions and 22 self-reported mTBI symptoms and find that five principal components from the manifolds (one component from the T1w manifold and four components from the DWI manifold) are significantly correlated with symptoms (p<0.05, uncorrected). The novelty of this work is that the DWI and T1w imaging metrics seem to contain information critical for distinguishing between mTBI and control subjects. This work presents an analysis of the pilot phase of data collection of the Quantitative Evaluation of Visual and Auditory Dysfunction and Multi-Sensory Integration in Complex TBI Patients study and defines specific hypotheses to be tested in the full sample.

Stroke is among the leading causes of death and disability throughout the world. Carotid atherosclerosis is a focal disease predominantly occurring at bifurcations, and for this reason, local progression/regression measurements of atherosclerosis allow for more sensitive detection of treatment effect than global measurements, such as total vessel wall volume (VWV). Vessel-wall-plus-plaque thickness (VWT) change has been developed to characterize local changes and has shown to be sensitive to treatment effect, but is unable to isolate changes in individual plaque components. In this work, we propose to quantify longitudinal voxel-by-voxel plaque-and-vessel-wall volume change (ΔVVol) and represent the ΔVVol distribution on a 3D standardized atlas. Such representation allows for quantitative comparison across patients and of the measurements obtained for the same patient at different time points. We introduced a 3D non-rigid registration framework to register the carotid ultrasound images acquired at baseline and a follow-up imaging session for each patient. A 3D volume equipped with voxel-by-voxel ΔVVol was obtained by taking the divergence of the displacement field obtained in non-rigid registration. This 3D volume was uniformly sampled in the vessel wall, and the ΔVVol distribution for each patient was represented in a 3D standardized map. The proposed 3D standardized ΔVVol map allows for the characterization of feature changes on a voxel-by-voxel basis that are masked in VWT quantification. This tool has the potential to further improve the sensitivity in treatment evaluation already attained by VWT quantification.

Glioblastoma multiforme (GBM) is the largest and most genetically and phenotypically heterogeneous category of primary brain tumors. Numerous novel chemical, targeted molecular and immune-active therapies in trial produce promising responses in a small disparate subset of patients but which patient will respond to which therapy remains unpredictable. Reliable imaging biomarkers for prediction and early detection of treatment response and survival are critical needs in neuro-oncology. In this study, brain tumor MRI 'deep features' extracted via transfer learning techniques were combined with features derived from an explicitly designed radiomics model to search for MRI markers predictive of overall survival (OS) in GBM patients. Two pre-trained convolutional neural network (CNN) models were utilized as the deep learning models and the elastic net-Cox model was performed to distinguish GBM patients into two survival groups. Two patient cohorts were included in this study. One was 50 GBM patients from our hospital and the other was 128 GBM patients from the Cancer Genome Atlas (TCGA) and the Cancer Image Archive (TCIA). The combined feature framework was predictive of OS in both data set with log-rank test p-value < 0.05 and may merit further study for reproducible prediction of treatment response.

Traditional deformable image registration (DIR) algorithms such as optical flow and demons are iterative and slow especially for large 4D-CT datasets. In order to quickly register the 4D-CT lung images for treatment planning and target definition, the computational speed of the current DIRs needs to be improved. Deep learning-based DIR methods that enable direct transformation prediction are promising alternatives for 4D-CT DIR. In this study, we propose to integrate dilated inception module (DIM) model and self-attention gate (Self-AG) into deep learning framework for 4DCT lung DIR. To overcome the shortage of manually aligned ‘ground truth’ training datasets, the network was designed to train in an unsupervised manner. Instead of using only the fixed and moving images as input, we also included the gradient images in x, y, z directions of the fixed and moving images as input to provide the network additional information to help the transformation prediction. The DIM was able to extract multi-scale structural features for robust feature learning. Self-AG were applied at different scales throughout the encoding and decoding pathways to highlight the structure representing feature differences between moving image and fixed image. The network was trained using pairs of 3D image patches that were extracted from any two random phases of one 4D-CT images. The loss function of the proposed network contains three parts which are image similarity loss, adversarial loss and a regularization loss. The network was trained and tested on 25 patients’ 4D-CT datasets using five-fold out cross validation. The proposed method was evaluated using Mean absolute error (MAE), peak signal to noise ratio (PSNR) and normalized cross correlation (NCC) between the deformed image and the fixed image. MAE, PSNR and NCC were 19.2±6.5, 35.4±3.0 and 0.995±0.002 respectively. Target registration errors (TREs) were calculated using manually selected landmark pairs. The average TRE was 3.38 ± 2.36 mm, which was comparable to traditional DIR algorithms. To summarize, the proposed method was able to achieve comparable performance to that of the traditional DIRs while being orders of magnitude faster (less than a minute).

CBCT has been widely integrated into modern linear accelerators in radiation therapy for image guidance purpose due to its cost effectiveness and low dose to patients. Planning CT images are registered with CBCT images for patient setup, contour propagations and dose calculations. However, it is challenging to accurately register the two since CBCT images often contain much image artifacts and noise, and intensity between CT and CBCT is not consistent. Therefore, traditional DIRs with intensity-based image similarity measures such as sum of squared differences, mean absolute differences are not applicable to CT-CBCT image registration. To address this issue, we propose to synthesize a high quality CT from CBCT to reduce image artifacts and perform intensity correction prior to image registration. Traditional Demons registration method was used to register the CT images with the CBCT-based synthetic CT (sCT). CT-sCT registration was tested on 5 patients’ datasets. On average, the mean absolute error between the fixed and deformed images were reduced from 102.2 to 92.4 HU by replacing CBCT with sCT.

Deformable registration problems are conventionally posed in a regularized optimization framework, where balance between fidelity and prescribed regularization usually needs to be manually tuned for each case. Even so, using a single weight to control regularization strength may be insufficient to reflect spatially variant tissue properties and limit registration performance. In this study, we propose to incorporate a spatially variant deformation prior into image registration framework using a statistical generative model. A generator network is trained in an unsupervised setting to maximize the likelihood of observing the moving and fixed image pairs, using an alternating back-propagation approach. The trained generative model imposes constraints on deformation and serves as an effective low dimensional deformation parametrization. During registration, optimization is performed over this learned parametrization, eliminating the need for explicit regularization and tuning. The proposed method was tested against a B-spline optimization method SimpleElastix, and an end-to-end learning method DIRNet. Experiment with synthetic images shows that our method yielded a registration error of (0.70±0.05) pixels, significantly lower than (0.86±0.12) pixels in SimpleElastix and (0.81±0.06) pixels in DIRNet. Experiment with 2D cardiac MR images demonstrates that the method completed registration with physically and physiologically more feasible deformations and the performance was close to the best of manually tuned results when evaluated with segmentation masks. The average registration time was 1.72 s, faster than 5.63 s in SimpleElastix.

Histology-based validation of magnetic resonance imaging (MRI) is dependent on spatial alignment of MRI and histology data obtained by very different physical techniques that usually generate very different types of image contrast. Most commonly, landmarks and/or mutual information (MI) techniques are employed for image coregistration. Landmark methods are limited by changed appearance or saliency of features manually selected in corresponding images. MI methods assume that pixel intensity distributions follow consistent relationships, however this assumption is often violated when differences exist in image texture and appearance. We propose a new approach based on local structure orientation (LSO). LSO-registration uses a tissue microstructure feature, fibre orientation, that can be measured by both histology and MRI, providing a dense feature field for an objective coregistration process. This structure feature, in-plane fibre orientation angle, can be generated from a histological image using a structure tensor analysis algorithm, and from 3D or 2D diffusion tensor MRI (DTI). We present a preliminary validation of LSO-registration based on digital and physical tissue structure phantoms that simulate fibre orientation in a human prostate. Using Elastix and Transformix we compare the performance of mutual information (MI), advanced mean squares (AMS), and a modified AMS method (AMSR) that corrects for angular redundancies. Registration accuracy was assessed quantitatively based on correction of deformed grid points and fiducial markers. Residual errors after registration using the physical tissue phantom with fiducial markers were: MI (3.4 ± 0.80), AMS (12.2 ± 5.7) and AMSR (2.1 ± 0.5).

Temporal subtraction of sequential chest radiographs based on image registration technique has been developed for decades to assist radiologists in the detection of interval changes. Although the performance of current methods is good, the computation cost of these methods is generally high. The high computation cost is mainly due to the iterative optimization problem of non-learning-based deformable registration. In this work we present a fast unsupervised learning-based algorithm for deformable registration of chest radiographs. Based on a convolutional neural network, the proposed model learns to directly estimate spatial transformations from pairs of moving images and fixed images, and uses the transformations to warp the moving images. We apply a regularization term to constrain the model to learn local matching. The model is trained by optimizing a pair-wise similarity metric between the warped moving image and the fixed image, with no need for any supervised information such as ground truth deformation fields. The trained model can be used to predict the warped moving images in one shot, and is thus very fast. The subtraction images of the warped images and the fixed images are able to enhance various interval changes. The preliminary results showed that for approximately 98.55% cases, the learning-based method could obtain improved or comparable registration in comparison with the baseline method.

In this study, we propose a new deep learning (DL) framework, a combination of fully convolutional and recurrent neural networks, and integrate them with a weakly supervised method for 3D MRI-transrectal ultrasound (TRUS) image registration. The MR and TRUS images are often highly anisotropic in its dimensions. For instance, in 3D US images the scale of each voxel in depth is often 5~10 times larger than that in each image slice. This high anisotropy makes the common 3D isotropic kernel suffer poor generality, resulting in unsatisfactory registration results in terms of accuracy. The key idea of the paper is to explicitly leverage 3D image anisotropy through the exploitation of the intra-slice context with a fully convolutional network (FCN) and the utilization of the inter-slice context with a recurrent neural network (RNN). After the 3D hierarchical features in MRI and TRUS have been extracted, we generate the dense deformation field by aligning corresponding prostate labels for individual image pairs. Experimental results showed that our proposed FCN-RNN neural network produces a mean target registration error (TRE) of 2.77±1.40 mm, and a mean dice similarity coefficient (DSC) of 0.9.

Spatial alignment of longitudinally acquired retinal images is necessary for the development of image-based metrics identifying structural features associated with disease progression in diseases such as age-related macular degeneration (AMD). This work develops and evaluates a feature-based registration framework for accurate and robust registration of retinal images. Methods: Two feature-based registration approaches were investigated for the alignment of fundus auto-fluorescence images. The first method used conventional SIFT local feature descriptors to solve for the geometric transformation between two corresponding point-sets. The second method used a deep-learning approach with a network architecture mirroring the feature localization and matching process of the conventional method. The methods were validated using clinical images acquired in an ongoing longitudinal study of AMD and consisted of 75 patients (145 eyes) with 4 year follow up imaging. In the deep-learning method, 113 image pairs were used during training (with the ground truth provided from manually verified SIFT feature registration) and 20 image pairs were used for testing (with the ground truth provided from manual landmark annotation). Results: Conventional method using SIFT features demonstrated target registration error (mean ± std) = 0.05 ± 0.04 mm, substantially improving the alignment from the initialization with error = 0.34 ± 0.22 mm. The deep-learning method, on the other hand, exhibited error = 0.10 ± 0.07 mm. While both methods improved upon the initial misalignment, SIFT method showed the best overall geometric accuracy. However, deep-learning method exhibited robust performance (error = 0.15 ± 0.09 mm) in the 7% of cases that SIFT method exhibited failures (error = 3.71 ± 6.36 mm). Conclusion: While both methods demonstrated successful performance, SIFT method exhibited the best overall geometric accuracy whereas deep-learning method was superior in terms of robustness. Achieving accurate and robust registration is essential in large-scale studies investigating factors underlying retinal disease progression such as in AMD.

The segmentation of bone, fat and muscle tissue in CT volumes is of special interest for surgeons and radiologists in order to diagnose some diseases and/or in surgery planning. These tissues are very difficult to delineate due to the presence of multiple and different structures and the similarity in terms of Hounsfield values with surrounding organs. In this paper, an automatic algorithm to implement the segmentation of bone, muscle and fat tissue is shown. The segmentation is carried out by minimizing an energy function via convex relaxation. In previous works carried out by the authors, only two labels had been considered (bone and muscle) and the methods had strong problems to segment skeletal muscle accurately due to the presence of internal organs with Hounsfield values similar to those within muscle tissue. In the present work, prior knowledge about the distribution of skeletal muscle in abdominal, chest and pelvis CT volumes has been considered by including a binary distance transform in the computation of the cost terms. Furthermore, a third label corresponding to fat tissue has been included in the segmentation. A public database has been used to assess the performance of the algorithm. Different metrics such as DICE, Jaccard, specificity, sensitivity and Positive Predictive Value (PPV) indexes have been obtained to evaluate the algorithm performance The technique has been compared with a previous multi-label scheme, a Hybrid level-set model implementation and a thresholding algorithm. The algorithm proposed outperformed the other methods in all the metrics considered.

The choroid is an important structure of the eye and choroid thickness distribution estimated from optical coherence tomography (OCT) images plays a vital role in analysis of many retinal diseases. This paper proposes a novel group-wise attention fusion network (referred to as GAF-Net) to segment the choroid layer, which can effectively work for both normal and pathological myopia retina. Currently, most networks perform unified processing of all feature maps in the same layer, which leads to not satisfactory choroid segmentation results. In order to improve this , GAF-Net proposes a group-wise channel module (GCM) and a group-wise spatial module (GSM) to fuse group-wise information. The GCM uses channel information to guide the fusion of group-wise context information, while the GSM uses spatial information to guide the fusion of group-wise context information. Furthermore, we adopt a joint loss to solve the problem of data imbalance and the uneven choroid target area. Experimental evaluations on a dataset composed of 1650 clinically obtained B-scans show that the proposed GAF-Net can achieve a Dice similarity coefficient of 95.21±0.73%.

Automatic lung segmentation with severe pathology plays a significant role in the clinical application, which can save physicians’ efforts to annotate lung anatomy. Since the lung has fuzzy boundary in low-dose computed tomography (CT) images, and the tracheas and other tissues generally have the similar gray value as the lung, it is a challenging task to accurately segment lung. How to extract key features and remove background features is a core problem for lung segmentation. This paper introduces a novel approach for automatic segmentation of lungs in low-dose CT images. First, we propose a contrastive attention module, which generates a pair of foreground and background attention maps to guide feature learning of lung and background separately. Second, a triplet loss is used on three feature vectors from different regions to pull the features from the full image and the lung region close whereas pushing the features from background away. Our method was validated on a clinical data set of 78 CT scans using the four-fold cross validation strategy. Experimental results showed that our method achieved more accurate segmentation results than that of state-of-the-art approaches.

This paper introduces an attention-guided channel to pixel convolution network for a fully automatic segmentation of retinal layers with choroidal neovascularization from optical coherence tomography (OCT) images. The proposed framework, consists of two new strategies for retinal layers segmentation: Channel to Pixel Block and Attention block. To deal with the contrast reduction of adjacent retinal layers caused by choroidal neovascularization, we firstly design a Channel to Pixel Block to convert particular channels into pixels in one bigger feature map, followed by a convolution layer optimized by a novel edge loss. Faced with large morphological changes of retinal layers, the attention mechanism is then introduced to extract more context information. The proposed method was trained on augmented 1280 OCT images and tested on 384 OCT images with choroidal neovascularization. The experimental results showed that the proposed method outperformed the state-of-art methods for retinal OCT image segmentation.

Accurate segmentation of pigment epithelial detachment (PED) in retinal optical coherence tomography (OCT) images can help doctors comprehensively analyze and diagnose chorioretinal diseases, such as age-related macular degeneration (AMD), central serous chorioretinopathy and polypoidal choroidal vasculopathy. Due to the serious uneven sizes of PED, some traditional algorithms or common deep networks do not perform well in PED segmentation. In this paper, we propose a novel attention multi-scale network (named as AM-Net) based on a U-shape network to segment PED in OCT images. Compared with the original U-Net, there are two main improvements in the proposed method: (1) Designing channel multiscale module (CMM) to replace the skip-connection layer of the U-Net, which uses channel attention mechanism to obtain multi-scale information. (2) Designing spatial multi-scale module (SMM) based on dilated convolution, which is inserted in the decoder path to make the network pay more attention on the multi-scale spatial information. We evaluated the proposed AM-Net on 240 clinically obtained OCT B-scans with 4-fold cross validation. The mean and standard deviation of Intersection over Union (IoU), Dice Similarity Coefficient (DSC), Sensitivity (Sen) and Specificity (Spe) are 72.12± 9.60%, 79.17±8.25%, 93.05±1.72% and 79.93±5.77%, respectively.

Abdominal multi-organ segmentation of computed tomography (CT) images has been the subject of extensive research interest. It presents a substantial challenge in medical image processing, as the shape and distribution of abdominal organs can vary greatly among the population and within an individual over time. While continuous integration of novel datasets into the training set provides potential for better segmentation performance, collection of data at scale is not only costly, but also impractical in some contexts. Moreover, it remains unclear what marginal value additional data have to offer. Herein, we propose a single-pass active learning method through human quality assurance (QA). We built on a pre-trained 3D U-Net model for abdominal multi-organ segmentation and augmented the dataset either with outlier data (e.g., exemplars for which the baseline algorithm failed) or inliers (e.g., exemplars for which the baseline algorithm worked). The new models were trained using the augmented datasets with 5-fold cross-validation (for outlier data) and withheld outlier samples (for inlier data). Manual labeling of outliers increased Dice scores with outliers by 0.130, compared to an increase of 0.067 with inliers (<0.001, two-tailed paired t-test). By adding 5 to 37 inliers or outliers to training, we find that the marginal value of adding outliers is higher than that of adding inliers. In summary, improvement on single-organ performance was obtained without diminishing multi-organ performance or significantly increasing training time. Hence, identification and correction of baseline failure cases present an effective and efficient method of selecting training data to improve algorithm performance.

Convolutional neural networks (CNNs) have been successfully applied to human brain segmentation. To in- corporate the left and right symmetry property of the brain into a network architecture, we propose a 3D left-right-reflection equivariant network to segment the anatomical structures of the brain. We extended previous group convolutions to account for left-right paired labels in the delineation. The proposed networks were compared with conventional networks trained with left-right reflection data augmentation in several tasks, showing improved performance. This is also the first work to extend reflection-equivariant CNNs to left-right paired labels in the human brain.

CBCT-guided adaptive prostate radiotherapy requires the accurate and rapid segmentation of the prostate and organs-at-risk from on-board CBCT images. Automatic segmentation based solely on CBCT can eliminate the need for registration to the planning CT, but has not been widely studied. Soft tissue contrast, such as prostate, is poor on CBCT images, which is often further degraded by CBCT artifacts. In this study, a new segmentation strategy is proposed for pelvic multi-organ segmentation aided by CBCT-based synthetic MRI. We tested our proposed method using 20 prostate cancer patients’ who underwent radiotherapy. The mean Dice similarity coefficient (DSC) of bladder, prostate and rectum were 95%, 86%, and 91%, which demonstrated the detection and segmentation accuracy of the proposed method. Methods for the accurate and fast segmentation of the prostate, bladder and rectum warrants further development in support a CBCT-guided adaptive radiotherapy workflow for prostate cancer.

In pharmaceutical research, optical coherence tomography (OCT) has been used for the assessment of diseases such as age-related macular degeneration (AMD) and retinal pigment epithelial (RPE) atrophy on animals in pre-clinical studies. To measure the thickness of the total retina and individual retina layers on these OCT images, it is necessary to perform accurate segmentation which is known to be a labor-intensive and error-prone task especially on images of diseased animals with significant retina distortion. Herein we elect to perform automated segmentation of retina layers on the OCT images of rodent subjects using deep convolutional neural networks (CNN). Based on a U-Net architecture, we perform segmentation of three most important retina layers using U-Net CNN models trained with three different strategies: Training from scratch, transfer learning, and continued training from a pre-trained model of a different animal cohort. To compare the three strategies, three models are trained and tested on OCT scans of rodent subjects, and the segmentation results are compared with manually corrected delineations using Dice similarity coefficient (DSC) as a measure of accuracy. Results show that although all three strategies lead to similar performance, transfer learning and continued training are effective in accelerating the training process, while continued training manages to generate the most accurate results that are also the most plausible via visual inspections.

Radiotherapy treatment is based on 3D anatomical models which require accurate organs-at-risk (OARs) delineation. In current clinical practice, the OARs are generally delineated from computed tomography (CT). Because of its superior soft-tissue contrast, magnetic resonance imaging (MRI) information can be introduced to improve the quality of these 3D OAR delineation and therefore the treatment plan itself. Manual segmentation of relevant tissue regions from MR image is a tedious and time-consuming procedure, which is also subject to inter- and intra-observer variation. In this work, we propose to use a 3D Faster R-CNN to automatically detect the locations of head and neck OARs, then utilize an attention U-Net to automatically segment the multiple OARs. We tested our method using 15 head and neck cancer patients. The mean Dice similarity coefficient (DSC) of esophagus, larynx, mandible, oral cavity, left parotid, right parotid, pharynx and spinal cord were 84%, 79%, 85%, 89%, 82%, 81%, 85% and 89%, which demonstrated the segmentation accuracy of the proposed U-Faster-RCNN method. This segmentation technique could be a useful tool to facilitate the routine clinical workflow of H&N radiotherapy.

Organoids are multicellular structures grown in the lab that resemble tissues or organs of the body. We recently generated human kidney organoids compatible with high throughput screening for developmental and disease phenotypes. Accurately segmenting large-scale image collections of organoids remains a challenge. We investigated automated segmentation of these structures using both conventional image processing algorithms and two different deep convolutional neural network architectures. Our dataset consisted of multi-channel images of organoids in 384-well plates, labeling distal tubules, proximal tubules, and podocytes as distinct segments. These images were used either for training and validation, or for testing. Each image was initially subjected to automated segmentation using a customized CellProfiler workflow. Separately, we performed semantic organoid segmentation using a Residual UNet (ResUNet) architecture, and instance organoid segmentation using a Mask R-CNN (MRCNN) architecture. For the latter, we compared model performance after initializing network weights in three different ways: randomly, using ResNet-50 weights pre-trained on the COCO dataset, and using ResUNet weights pre-trained on organoid images. Using ResUNet or randomly initializing MRCNN backbone weights provided improved semantic segmentation compared to using precomputed weights from COCO or ResUNet, or to using the CellProfiler workflow. Conversely, using precomputed weights to initialize MRCNN provided better instance segmentation accuracy and sensitivity than random initialization. Our findings provide a basis for automated segmentation of organoids with convolutional neural networks, to aid in high throughput screening for compounds relevant to renal phenotypes.

Monitoring retinal thickness of persons with multiple sclerosis (MS) provides important bio-markers for disease progression. However, changes in retinal thickness can be small and concealed by noise in the acquired data. Consistent longitudinal retinal layer segmentation methods for optical coherence tomography (OCT) images are crucial for identifying the real longitudinal retinal changes of individuals with MS. In this paper, we propose an iterative registration and deep learning based segmentation method for longitudinal 3D OCT scans. Since 3D OCT scans are usually anisotropic with large slice separation, we extract B-scan features using 2D deep networks and utilize inter-B-scan context with convolutional long-short-term memory (LSTM). To incorporate longitudinal information, we perform fundus registration and interpolate the smooth retinal surfaces of the previous visit to use as a prior on the current visit.

Accurate and reproducible tissue identification techniques are essential for understanding structural and functional changes that either occur naturally with aging, or because of chronic disease, or in response to intervention therapies. These image analysis techniques are frequently utilized for characterization of changes in bone architecture to assess fracture risk, and for the assessment of loss of muscle mass and strength defined as sarcopenia. Peripheral quantitative computed tomography (pQCT) is widely employed for tissue identification and analysis. Advantages of pQCT scanners are compactness, portability, and low radiation dose. However, these characteristics imply limitations in spatial resolution and SNR. Therefore, there is still a need for segmentation methods that address image quality limitations and artifacts such as patient motion. In this paper, we introduce multi-atlas segmentation (MAS) techniques to identify soft and hard tissues in pQCT scans of the proximal tibia (~ 66% of tibial length) and to address the above factors that limit delineation accuracy. To calculate the deformation fields, we employed multi-grid free-form deformation (FFD) models with B-splines and a symmetric extension of the log-domain diffeomorphic demons (SDD). We then applied majority voting and Simultaneous Truth And Performance Level Estimation (STAPLE) for label fusion. We compared the results of our MAS methodology for each deformable registration model and each label fusion method, using Dice similarity coefficient scores (DSC). The results show that our technique utilizing SDD with STAPLE produces very good accuracy (DSC mean of 0.868) over all tissues, even for scans with considerable quality degradations caused by motion artifacts.

Sensitive and cost-effective biomarkers for carotid atherosclerosis are required to evaluate the efficacy of dietary and medical treatments. Carotid atherosclerosis is a focal disease predominantly occurring in bends and bifurcations. For this reason, we have previously developed a method to measure local vessel-wall-plus-plaque thickness (VWT); a biomarker based on a weighted average of the point-wise ΔVWT was also developed and validated to be sensitive to treatment effect. However, the weight determined on a point-by-point basis did not take into account the spatial correlation of ΔVWT in neighboring points. In this paper, we developed a biomarker that is able to characterize the correlation within each local patch of the VWT map. The deep autoencoder (DAE) initialized by the stacked restricted Boltzmann machines (RBMs) was introduced to learn a compact feature representation of each patch in the 2D VWT map. The patch-based feature change was obtained by taking the difference between the features obtained at baseline and a follow-up imaging session. The new biomarker, denoted by ∆VWT_patch, was computed by taking a weighted average of the patch-based feature change. The sensitivity of the patch-based average was compared with that of the point-wise average (∆VWT_point) in 40 subjects involved in a placebo-controlled clinical trial of the efficacy of pomegranate. ∆VWT_patch detected a significant difference between the pomegranate and placebo groups (p = 0.017), but not ∆VWT_point (p = 0.056). The sample size required by ∆VWT_patch to establish significance was 37% smaller than that by ∆VWT_point.