This PDF file contains the front matter associated with SPIE Proceedings Volume 11598, including the Title Page, Copyright information, and Table of Contents.

Welcome and Introduction to SPIE Medical Imaging conference 11598: Image-Guided Procedures, Robotic Interventions, and Modeling

The mitral valve (MV) regulates blood flow into the left ventricle (LV). In situations where the MV fails to fully close the resulting blood regurgitation into the left atrium causes pulmonary congestion, leading to heart failure and/or stroke. There is now agreement that adjunctive procedures are required to treat IMR, yet there is no consensus regarding the best procedure. There is thus an urgent need for quantitative assessments of MV function to better design surgical solutions. We demonstrate state-of-the-art means to produce patient-specific MV computational models to develop quantitatively optimized devices and procedures for MV repair that incorporate LV geometry.

Prostate cancer is the most frequently diagnosed non-cutaneous cancer and the second leading cause of cancer-related deaths in men. Whole gland surgical and radiation treatments for prostate cancer are highly effective for long-term cancer control. However, these are often associated with overtreatment, resulting in urinary complications and sexual dysfunction, adversely impacting the quality of life. Focal laser ablation (FLA) under magnetic resonance imaging (MRI)-guidance is an alternative minimally invasive treatment method for localized prostate tumors while preserving surrounding structures and healthy tissues. Accurate needle positioning and delivery are critical for the therapeutic success of MRI-guided FLA. We propose an MRI-compatible mechatronic system for in-bore transperineal FLA needle guidance to localized prostate lesions. This paper presents the mechatronic system design, including a remotely actuated, four degree-of-freedom transperineal positioning and needle guidance mechanism, and adaptable needle guide. We demonstrate its MR compatibility and evaluated its mechanical bias in free-space testing using an external optical tracking system with several measurement points (N=40) over its range-of-motion. Free-space testing resulted in a root-mean-square error of 0.71 ± 0.30 mm. Within an MR environment, in-bore testing to virtual targets (N=10) with projected needle trajectories resulted in a mean needle tip error of 1.81 ± 0.56 mm and needle trajectory error of 0.78 ± 0.75°. This suggests that localized ablation regions can be accurately targeted within 2.16 mm within 95% confidence. An extensive in-bore analysis and correction for systematic bias across the range-of-motion may improve this accuracy. This study shows that our proposed mechatronic needle guidance system may be a feasible alternative for accurate MR-guided FLA for localized prostate therapy.

Over the past several years, researchers have made significant progress toward providing image guidance for the da Vinci system, utilizing data sources such as robot kinematic data, endoscope image data, and preoperative medical images. One data source that could provide additional subsurface information for use in image guidance is the da Vinci’s FireFly camera system. FireFly is a fluorescence imaging feature for the da Vinci system that uses injected indocyanine green dye and special endoscope filters to illuminate subsurface anatomical features as the surgeon operates. FireFly is now standard for many surgical procedures with the da Vinci robot; however, it is currently challenging to understand spatial relationships between pre-operative CT images and intraoperative fluorescence images. Here, we extend our image guidance system to incorporate FireFly information as well, so that the surgeon can view FireFly data in the image guidance display while operating with the da Vinci robot. We present a method for reconstructing 3D models of the FireFly fluorescence data from endoscope images and mapping the models into an image guidance display that also includes segmented, registered preoperative CT images. We analyze the accuracy of our reconstruction and mapping method and present a proof-of-concept application where we reconstruct a fluorescent subsurface blood vessel and map it into our image guidance display. Our method could be used to provide surgeons with additional context for the FireFly fluorescence imaging data or to provide additional data for computing or verifying the registration between the robot and preoperative images.

Purpose: Pelvic fracture fixation is a challenging procedure that commonly relies on 2D fluoroscopic guidance to place guidewires within complex bone corridors. Prior work reported on a video-on-drill navigation system for guidewire insertion as a potential solution. Here, we assess performance across a range of hardware components to help guide the design of future system prototypes with respect to clinical requirements. Methods: The video-on-drill system uses a camera rigidly mounted on the drill and multimodality fiducial markers (optical and radio-opaque) to provide real-time trajectory visualization. This work reports on the selection of a new camera+lens configuration. Configurations were assessed across two cameras (referred to as the ArduCam and ELP) and five lens options (A45-A90). Clinical requirements were specified by an orthopaedic-surgeon in terms of the nominal drill operating distance (𝐷) and operating area (𝐴). Performance was evaluated in terms of the accuracy of fiducial marker pose estimation (δ_𝐷), and the field of view (FOV). Results: At matched FOV, the accuracy for the ELP camera was significantly better (p < 0.01) with median δ_𝐷 of 1.26 mm (1.1 mm IQR) compared to the ArduCam [median δ_𝐷 of 1.85 mm (2.0 mm IQR)]. The accuracy of the A45 and A55 lens was found to be suitable (δ_𝐷 < 2 mm) while providing sufficient FOV at nominal drill operating conditions. Conclusion: With respect to application requirements, the camera+lens combination (ELP+Ardu55) was identified to provide the best performance, serving as an important precursor to future design iterations of the video-on-drill system.

Currently-available metallic retractors typically used in transoral robotic surgery (TORS) cause significant artifacts in CT imaging and cannot be safely used in MRI. The lack of imaging-compatible oral retractors poses a significant challenge to enabling intraoperative imaging in TORS. This work introduces a customizable compact 3D-printed polymer retractor system that enables multiple modes of adjustability, artifact-free CT and MR images, and adequate surgical exposure. The polymer retractor design was modeled after the traditional metal FK and Crowe-Davis retractors and can be used with an acrylic suspension system that rests over the patient’s chest. Finite element analysis was conducted to evaluate the mechanical performance in relevant clinical loading conditions. Cadaver experiments followed by endoscopic, CT, and MR imaging were performed to demonstrate functionality. Artifact-free CT and MR images were obtained. An interincisive distance of 42.50 mm and 200.09 cm³ working volume were achieved, which allow the introduction of robotic arms and necessary instruments in TORS. This polymer retractor system makes it possible to acquire intraoperative images and establishes a critical step to make image-guided TORS both feasible and effective.

Image guidance for abdominal procedures requires an anatomical model capable of representing significant displacement and deformation of relevant tissues in a computationally efficient manner. This work evaluates the suitability of statistical shape modeling to represent key structures in Robot Assisted Laparoscopic Partial Nephrectomy (RALPN) both individually, and also as a multi-body composite model. Tomography obtained from subjects in an ongoing RALPN study was used to produce surface model representations of the kidneys, abdominal aorta, and inferior vena cava. Each structure was resliced and remeshed in a standardized fashion to allow for extraction of the principal modes of variation. Reduced parameter representations of the example structures based on the strongest eigenmodes indicate that <5mm average RMSE modeling accuracy can be achieved with four parameters for the individual models and eight parameters for the four-body composite model. The magnitude of centroid displacements observed under the principal modes of variation is consistent with literature-reported values, suggesting that this approach may be suitable for image guidance in RALPN.

Purpose: Emerging deep-brain stimulation (DBS) procedures require a high degree of accuracy in placement of neuroelectrodes, even in the presence of deformation due to cerebrospinal fluid (CSF) egress during surgical access. We are developing ventriculoscope and hand-eye calibration methods for a robot-assisted guidance system to augment accurate electrode placement through transventricular approach. Methods: The ventriculoscope camera was modelled and calibrated for lens distortion using three different checkerboards, followed by evaluation on a separate board. The experimental system employed a benchtop UR3e robot (Universal Robots, Denmark) and ventriculoscope (Karl Storz, Tuttlingen, Germany) affixed to the end effector – referred to as the robotassisted ventriculoscopy (RAV) platform. Performance was evaluated in terms of three error metrics (RPE, FCE and PDE). Experiments were conducted to estimate the camera frame of reference using hand-eye calibration methods, and evaluated using a ChAruco board, using five different solvers and residual calibration error as the metric. Results: Camera calibration demonstrated subpixel (0.81 ± 0.11) px reprojection error and projection distance error (PDE) <0.5 mm. The error was observed to converge for any checkerboard used given a sufficient number of calibration images. The hand-eye calibration exhibited sub-mm residual error (0.26 ± 0.18) mm insensitive to the solver used. Conclusions: The RAV system demonstrates sub-mm ventriculoscope camera calibration error and robot-to-camera handeye residual error, providing a valuable platform for the development of advanced 3D guidance systems for emerging DBS approaches. Future work aims to develop structure-from-motion (SfM) methods to reconstruct a 3D optical scene using endoscopic video frames and further testing using rigid and deformable anatomical phantoms as well as cadaver studies.

Surgical tool segmentation is becoming imperative to provide detailed information during intra-operative execution. These tools can obscure surgeons’ dexterity control due to narrow working space and visual field-of-view, which increases the risk of complications resulting from tissue injuries (e.g. tissue scars and tears). This paper demonstrates a novel application of segmenting and removing surgical instruments from laparoscopic/endoscopic video using digital inpainting algorithms. To segment the surgical instruments, we use a modified U-Net architecture (U-NetPlus) composed of a pre-trained VGG11 or VGG16 encoder and redesigned decoder. The decoder is modified by replacing the transposed convolution operation with an up-sampling operation based on nearest-neighbor (NN) interpolation. This modification removes the artifacts generated by the transposed convolution, and, furthermore, these new interpolation weights require no learning for the upsampling operation. The tool removal algorithms use the tool segmentation mask and either the instrument-free reference frames or previous instrument-containing frames to fill-in (i.e., inpaint) the instrument segmentation mask with the background tissue underneath. We have demonstrated the performance of the proposed surgical tool segmentation/removal algorithms on a robotic instrument dataset from the MICCAI 2015 EndoVis Challenge. We also showed successful performance of the tool removal algorithm from synthetically generated surgical instruments-containing videos obtained by embedding a moving surgical tool into surgical tool-free videos. Our application successfully segments and removes the surgical tool to unveil the background tissue view otherwise obstructed by the tool, producing visually comparable results to the ground truth.

Computer-Assisted Surgery (CAS) aids the surgeon by enriching the surgical scene with additional information in order to improve patient outcome. One such aid may be the superimposition of important structures (such as blood vessels and tumors) over a laparoscopic image stream. In liver surgery, this may be achieved by creating a dense map of the abdominal environment surrounding the liver, registering a preoperative model (CT scan) to the liver within this map, and tracking the relative pose of the camera. Thereby, known structures may be rendered into images from the camera perspective. This intraoperative map of the scene may be constructed, and the relative pose of the laparoscope camera estimated, using Simultaneous Localisation and Mapping (SLAM). The intraoperative scene poses unique challenges, such as: homogeneous surface textures, sparse visual features, specular reflections and camera motions specific to laparoscopy. This work compares the efficacies of two state-of the-art SLAM systems in the context of laparoscopic surgery, on a newly collected phantom dataset with ground truth trajectory and surface data. The SLAM systems chosen contrast strongly in implementation: one sparse and feature-based, ORB-SLAM3,1{3 and one dense and featureless, ElasticFusion.4 We find that ORB-SLAM3 greatly outperforms ElasticFusion in trajectory estimation and is more stable on sequences from laparoscopic surgeries. However, when extended to give a dense output, ORB-SLAM3 performs surface reconstruction comparably to ElasticFusion. Our evaluation of these systems serves as a basis for expanding the use of SLAM algorithms in the context of laparoscopic liver surgery and Minimally Invasive Surgery (MIS) more generally.

This paper proposes a novel self-supervised depth estimation method guided by a context encoder. Depth estimation from stereo laparoscopic images is essential to robotic surgical navigation systems and robotic surgical platform. Recent work has shown that depth estimation of stereo image pairs can be formulated as a self- supervised learning task without ground-truth. However, most architectures based on convolutional neural lead to lose some spatial information because of the consecutive pooling and convolution operations. In order to tackle this problem, we add a contextual encoding module to the previous method. The context encoder module is formed by dense atrous convolution block and spatial pyramid pooling block that are used to extract and merge features on different scales. Also, we add the edge-awared smoothness for predicted disparity maps. In addition, we output multi-scale disparity predictions and corresponding image reconstruction for loss calculating. In the experiments, we showed that the proposed method has about 7.79% improvement in SSIM and about 17.76% improvement in PSNR for stereo image pairs compared with previous method. Also, the disparity maps and reconstructed images given by the proposed method have significant enhancements compared with the previous method.

Preoperative CT (pCT) images acquired in a supine position can be inaccurate for intraoperative image guidance in open spine surgery due to alignment change between supine pCT and intraoperative prone positioning. We have developed a level-wise registration framework to compensate for the alignment change using intraoperative stereovision (iSV) data of the exposed spine. A hand-held stereovision system was developed, but the field of view was limited to 1-2 segments per image. Although multiple iSV surfaces can be combined to capture the full field based on tracking information, acquisition is limited to one snapshot at a time with minimized hand motion due to asynchronization between image and tracking data. In this study, we developed methods to concatenate iSV surfaces without relying on tracking information, and illustrate the methods using data acquired from a pig spine. To register two iSV surfaces, the 2D texture maps were registered using an optical flow algorithm, and the 3D point cloud of the second iSV surface was registered with the first using 3D spatial information of each pixel. The two registered iSV surfaces were then merged to form one composite iSV surface. Multiple iSV surfaces were stitched sequentially. Results from 4 image pairs show that 2D and 3D registration accuracy was 2.7±0.6 pixels and 1.0±0.1 mm, respectively, across 8 landmarks. The overall accuracy of the final composite surface was 0.9±0.4 mm. These preliminary data show that iSV can potentially be acquired at video rate to improve efficiency to recover the full surgical field.

We propose a method for the three-dimensional reconstruction from a single-shot colonoscopic image. Extracting a three-dimensional colon structure is an important task in colonoscopy. However, a colonoscope captures only two-dimensional information as colonoscopic images. Therefore, an estimation of three-dimensional information from two-dimensional images has potential demands. In this paper, we integrate deep-learning-based depth estimation to three-dimensional reconstruction. This approach omits the inaccurate corresponding matching from the procedure of conventional three-dimensional reconstruction. We experimentally demonstrated accurate reconstructions with comparisons between a polyp size in three-dimensional reconstruction and an endoscopist's measurement.

Late gadolinium enhanced (LGE) cardiac magnetic resonance (CMR) imaging, the current benchmark for assessment of myocardium viability, enables the identification and quantification of the compromised myocardial tissue regions, as they appear hyper-enhanced compared to the surrounding, healthy myocardium. However, in LGE CMR images, the reduced contrast between the left ventricle (LV) myocardium and LV blood-pool hampers the accurate delineation of the LV myocardium. On the other hand, the balanced-Steady State Free Precession (bSSFP) cine CMR imaging provides high resolution images ideal for accurate segmentation of the cardiac chambers. In the interest of generating patient-specific hybrid 3D and 4D anatomical models of the heart, to identify and quantify the compromised myocardial tissue regions for revascularization therapy planning, in our previous work, we presented a spatial transformer network (STN) based convolutional neural network (CNN) architecture for registration of LGE and bSSFP cine CMR image datasets made available through the 2019 Multi-Sequence Cardiac Magnetic Resonance Segmentation Challenge (MS-CMRSeg). We performed a supervised registration by leveraging the region of interest (RoI) information using the manual annotations of the LV blood-pool, LV myocardium and right ventricle (RV) blood-pool provided for both the LGE and the bSSFP cine CMR images. In order to reduce the reliance on the number of manual annotations for training such network, we propose a joint deep learning framework consisting of three branches: a STN based RoI guided CNN for registration of LGE and bSSFP cine CMR images, an U-Net model for segmentation of bSSFP cine CMR images, and an U-Net model for segmentation of LGE CMR images. This results in learning of a joint multi-scale feature encoder by optimizing all three branches of the network architecture simultaneously. Our experiments show that the registration results obtained by training 25 of the available 45 image datasets in a joint deep learning framework is comparable to the registration results obtained by stand-alone STN based CNN model by training 35 of the available 45 image datasets and also shows significant improvement in registration performance when compared to the results achieved by the stand-alone STN based CNN model by training 25 of the available 45 image datasets.

Proton beam therapy has the potential to non-invasively treat ventricular tachycardia (VT) by homogenizing infarct scar. It has been previously demonstrated that proton beam therapy can create lesions in healthy myocardial tissue, thereby suggesting a potential for treatment of VT. In prior work, we quantified the relationship between dose delivered to myocardial tissue with lesion formation identified with in vivo, delayed contrast-enhanced magnetic resonance imaging (DCE-MRI) scans. In the current work, we evaluate the relationship of delivered dose with lesions identified in high resolution, post-mortem DCE-MRI scans. Deformable registration is used to align the dose maps from the baseline planning CT scans to ex vivo scans following proton beam therapy in swine. The current study demonstrates that nearly 100% of tissue exposed to a dose of 30 Gy or higher developed into lesion and approximately 85% of the tissue in the 20-30 Gy interval developed into lesion. On the other hand, tissue exposed to doses of 10 Gy or less tended to remain healthy myocardium, with less than 10% of tissue in the 5-10 Gy range and almost no tissue in the 0-5 Gy range developing into lesion.

The definitive diagnosis of breast cancer still requires obtaining a tissue sample, most commonly with an image-guided needle biopsy. High-resolution breast-dedicated positron emission mammography (PEM) has been used to guide biopsy due to its high sensitivity, specificity, and positive predictive value in detecting lesions. Although few promising methods have been proposed, these approaches require iterative procedures to verify three-dimensional (3D) needle positioning and anatomical context are not yet available. To address these challenges, we previously developed a mechatronic guidance system integrating an ultrasound (US)-guided core-needle biopsy (CNB) intervention to operate with a high-resolution PEM system. This paper presents the first assessment of system performance and phantom validation for PEM-US-guided biopsy under high-resolution PEM localization. We optimized the user interface by implementing a dynamic visualization module for 3D guidance within a simulated PEM image. Breast phantoms with targets in known 3D positions were fabricated, simulating PEM localized breast lesions and corresponding to virtual targets in the PEM image. The mechatronic guidance system was registered to the simulated PEM detector plate and real-time targeting was performed in three spatial dimensions. Biopsy procedures (N=8) were 100% successful in targets 6 mm diameter and height with a mean 3D positioning error <1.0 mm, and 0.72 mm in-plane and 0.34 mm cross-plane component errors, relative to the PEM image coordinate space. This first proof-of-concept study demonstrates a highly accurate system and feasible workflow toward PEM-USguided biopsy under high-resolution PEM localization

It is imperative that image-guided intervention (IGI) systems provide accurate and precise navigation information to enable the user to trust the system and not place unwarranted confidence in the guidance capabilities of the system. Unfortunately, the actual error associated with the overall targeting capabilities of an IGI system is not readily known. Here we are primarily interested in the application of image-guided surgery in the context of renal interventions. We built a simulation pipeline to study the uncertainty propagation through an optically tracked IGI system to gain insight into the overall accuracy of the system. Our simulation pipeline models several stages, including stylus calibration, tool tracking, patient tracking, and image to patient registration. In the effort to realistically estimate tracking noise and user-associated fiducial localization error (FLE), we conducted several experiments using the optical tracking system. Our simulation suggested that a wider cone angle results in a more accurate tool calibration, which improves further with the collection of additional samples. Furthermore, our simulations also suggested that the image-to-patient registration was the most significant contributor to navigation uncertainty, followed by the fiducial localization error. Lastly, we also observed a 0.72 correlation between the Target Registration Error (TRE) estimated at target fiducials and the distance between the the centroids of the registration and target fiducial landmarks. To validate the simulation predictions, we also conducted several in vitro experiments using a 3D printed patient specific kidney phantom and compared the simulation-based registration predictions with those observed experimentally in vitro. The experiments confirmed the registration metrics (Fiducial Registration Error and TRE) predicted by the simulations, given several specific combinations of fiducial landmarks used to perform the image to patient registration.

Although MRI with hepatospecific contrast agents is a new standard diagnostic imaging for patients with liver cancer, there are no automated methods for detailed segmentation of the liver vasculature in cases with progressed tumors. This is due to the anatomical complexity, underlying disease and tunability of MRI image contrast, which challenge automatization. Here, we investigated the feasibility of liver vessel segmentation with three CNN architectures in combination with four different loss functions. In particular, a 3D Unet, a Vnet and its modification with an intra-layer dense block (DVnet) were evaluated. Dice-based loss, categorical cross entropy (CCE), weighed categorical cross entropy (WCCE) and focal loss (FL) were used as loss functions for training, the latter two to deal with the imbalanced class problem. A cohort of 90 patients (60 training, 10 validation and 20 testing) with progressed liver tumors were involved in this study, with manually annotated liver vasculature as a “gold standard”. Trained networks were evaluated by means of the Dice coefficient and centerline-based F1 score calculations. Models trained with balanced loss functions (FL,WCEE) performed the best for DVnet, while Vnet had the best performance for unbalanced loss functions. Vnet and DVnet architectures trained with an FL had the best overall segmentation accuracy (DC = 70%), while networks with a Dicebased loss had the lowest performance (max DC = 42%). In conclusion, the use of balanced loss functions, addition of an intra-layer dense block and drop-outs into the network architecture improved handling the unbalanced class problem in liver vessel segmentation.

Deep brain stimulation (DBS) has been recently approved by the FDA to treat epilepsy patients with refractory seizures, i.e., patients for whom medications are not effective. It involves stimulating the anterior nucleus of the thalamus (ANT) with electric impulses using permanently placed electrodes. One main challenge with the procedure is to determine a trajectory to place the implant at the proper location while avoiding sensitive structures. In this work, we focus on one category of sensitive structures, i.e., brain vessels, and we propose a method to segment them in clinically acquired contrast-enhanced T1-weighted (T1CE) MRI images. We develop a deep-learning-based 3D U-Net model that we train/test on a set of images for which we have created the ground truth. We compare this approach to a traditional vesselness-based technique and we show that our method produces significantly better results (Dice score: 0.794), especially for small vessels.

Accurate segmentation of the prostate on computed tomography (CT) has many diagnostic and therapeutic applications. However, manual segmentation is time-consuming and suffers from high inter- and intra-observer variability. Computerassisted approaches are useful to speed up the process and increase the reproducibility of the segmentation. Deep learningbased segmentation methods have shown potential for quick and accurate segmentation of the prostate on CT images. However, difficulties in obtaining manual, expert segmentations on a large quantity of images limit further progress. Thus, we proposed an approach to train a base model on a small, manually-labeled dataset and fine-tuned the model using unannotated images from a large dataset without any manual segmentation. The datasets used for pre-training and finetuning the base model have been acquired in different centers with different CT scanners and imaging parameters. Our fine-tuning method increased the validation and testing Dice scores. A paired, two-tailed t-test shows a significant change in test score (p = 0.017) demonstrating that unannotated images can be used to increase the performance of automated segmentation models.

During tumor resection surgery, intraoperative ultrasound images of the brain show anatomical structures, the tumor, and the resection cavity (after resection started). These elements help with the localization and tumor resection, and can be used to register the preoperative MRI to intraoperative images, to compensate for the tissue deformation occurring during surgery. We evaluate a multi-class segmentation model for the sulci, falx cerebri, tumor, resection cavity and ventricle. We present strategies to overcome the severe class imbalance in the training data. We show that a multi-class model may leverage inter-class spatial relationships and produce more accurate results than single-class models.

This paper describes a bronchial orifice (BO) segmentation method on real bronchoscopic video frames by using depth images. The BO is one of the anatomical characteristics in the bronchus, which is critical in clinical applications such as bronchus scene description and navigation path generation. Previous work used image appearance and the gradation of the real bronchoscopic image to segment orifice region, which behaved poorly in complex scenes including bubble or changes in illumination. To obtain a better segmentation result of BO even in the complex scenes, we propose a BO segmentation method using the distance between the bronchoscope camera and the bronchus lumen, which is represented by a depth image. Since the depth image is unavailable due to devices limitation, we use an image-to-image domain translation network named cycle generative adversarial network (CycleGAN) to estimate depth images from real bronchoscopic images. The BO regions are considered as the regions whose distances are larger than a distance threshold. We decide the distance threshold according to the depth images' projection profiles. Experimental results showed that the proposed method can find BO regions in the real bronchoscopic videos in real-time. We manually labeled BO regions as ground truth to evaluate the proposed method. The average Dice score of the proposed method was 77.0 %.

Accurate and efficient 3D catheter navigation within the airways is crucial for transbronchial procedures, including biopsy and tumor ablation. While electromagnetic tracking for 3D tip localization exists, it requires specialized equipment, timeconsuming registration steps and is prone to CT-to-body divergence. Recently developed techniques allow 3D reconstruction of catheters and curvilinear devices from two simultaneously acquired projection images, but they require a biplane C-arm x-ray system which is not widely available in clinical practice. This study investigates a method of timeresolved 3D tracking of catheters using the more widely available single-plane C-arm system. Imaging was performed using an acquisition protocol where the C-arm continuously rotates back and forth within a limited angular range while acquiring x-ray images. The catheter reconstruction was performed using a sliding window of the most recent x-ray images, which captures information on device shape and position versus time. A model-based approach was used to estimate the catheter shape and position at the time of the last x-ray image acquisition. To evaluate the approach, a pig study was performed where the proposed reconstruction was compared to a gold standard extracted from cone-beam CT (CBCT). The average 3D root mean squared distance between single plane and CBCT reconstruction was 0.8 ± 0.3mm for a stationary catheter and 2.4 ± 1.4mm for a catheter moving at ~1 cm/s. The tip localization error ranged from 1.0 ± 0.4mm to 3.8 ± 2.2mm. The results indicate catheter navigation based on the proposed single plane C-arm imaging technique is feasible with reconstruction errors on the order of the typical ablation catheter diameter (2.0 - 3.2mm).

In this study, we introduce statistical modeling methods for pneumothorax deformation using paired cone-beam computed tomography (CT) images. We designed a deformable mesh registration framework for shape changes involving non-linear deformation and rotation of the lungs. The registered meshes with local correspondences are available for both surgical guidance in thoracoscopic surgery and building statistical deformation models with inter-patient variations. In addition, a kernel-based deformation learning framework is proposed to reconstruct intraoperative dfl ated states of the lung from the preoperative CT models. This paper reports the findings of pneumothorax deformation and evaluation results of the kernel-based deformation framework.

Quantitative thoracic dynamic magnetic resonance imaging (QdMRI), a recently developed technique, provides a potential solution for evaluating treatment effects in thoracic insufficiency syndrome (TIS). In this paper, we demonstrate how lung parenchymal characteristics can be assessed via intensity properties in lung dynamic MRI, a modality suitable for use in pediatric patients. The QdMRI-based approach includes dynamic MR image acquisition, 4D image construction, image pre-processing with non-uniformity correction and intensity standardization, and lung segmentation from the 4D constructed image via a deep learning approach, as well as extraction of image parenchymal intensity properties from the segmented lungs and statistical comparisons among different clinical scenarios. We include 22 dMRI scans from 11 TIS patients (each with both pre-operative and post-operative scans) and 23 dMRI scans from healthy children. Two-sided paired t-testing is performed to compare lung intensity properties between end of expiration (EE) and end of inspiration (EI) within TIS patients (pre-operative and post-operative, separately) and normal children. We also compare the lung intensity properties at EE and EI among pre-operative TIS patients, post-operative TIS patients, and normal children. Experimental results show that lung (T2) intensity at EI is significantly lower than that at EE and lung intensity of postoperative TIS patients is significantly lower than that in pre-operative TIS patients and closer to that of normal children than to that of pre-operative TIS patients, indicating improvement in lung aeration. To our knowledge, this is the first study to provide a quantitative dynamic functional method to analyze lung parenchyma during tidal breathing on dynamic MRI in both healthy children and pediatric patients with TIS.

We describe a novel, two-stage computer assistance system for lung anomaly detection using ultrasound imaging in the intensive care setting to improve operator performance and patient stratification during coronavirus pandemics. The proposed system consists of two deep-learning-based models: a quality assessment module that automates predictions of image quality, and a diagnosis assistance module that determines the likelihood-of-anomaly in ultrasound images of sufficient quality. Our two-stage strategy uses a novelty detection algorithm to address the lack of control cases available for training the quality assessment classifier. The diagnosis assistance module can then be trained with data that are deemed of sufficient quality, guaranteed by the closed-loop feedback mechanism from the quality assessment module. Using more than 25,000 ultrasound images from 37 COVID-19-positive patients scanned at two hospitals, plus 12 control cases, this study demonstrates the feasibility of using the proposed machine learning approach. We report an accuracy of 86% when classifying between sufficient and insufficient quality images by the quality assessment module. For data of sufficient quality – as determined by the quality assessment module – the mean classification accuracy, sensitivity, and specificity in detecting COVID-19-positive cases were 0.95, 0.91, and 0.97, respectively, across five holdout test data sets unseen during the training of any networks within the proposed system. Overall, the integration of the two modules yields accurate, fast, and practical acquisition guidance and diagnostic assistance for patients with suspected respiratory conditions at pointof- care.

Lobectomy is a common and effective procedure for treating early-stage lung cancers. However, for patients with compromised pulmonary function (e.g. COPD) lobectomy can lead to major postoperative pulmonary complications. A technique for quantitatively predicting postoperative pulmonary function is needed to assist surgeons in assessing candidate’s suitability for lobectomy. We present a framework for quantitatively predicting the postoperative lung physiology and function using a combination of lung biomechanical modeling and machine learning strategies. A set of 10 patients undergoing lobectomy was used for this purpose. The image input consists of pre- and post-operative breath hold CTs. An automated lobe segmentation algorithm and lobectomy simulation framework was developed using a Constrained Adversarial Generative Networks approach. Using the segmented lobes, a patient-specific GPU-based linear elastic biomechanical and airflow model and surgery simulation was then assembled that quantitatively predicted the lung deformation during the forced expiration maneuver. The lobe in context was then removed by simulating a volume reduction and computing the elastic stress on the surrounding residual lobes and the chest wall. Using the deformed lung anatomy that represents the post-operative lung geometry, the forced expiratory volume in 1 second (FEV1) (the amount of air exhaled by a patient in 1 second starting from maximum inhalation), and forced vital capacity (FVC) (the amount of air exhaled by force from maximum inhalation), were then modeled. Our results demonstrated that the proposed approach quantitatively predicted the postoperative lobe-wise lung function at the FEV1 and FEV/FVC.

To perform lung cancer staging, the physician uses bronchoscopy to stage the central-chest lymph nodes. Because the lymph nodes lie outside the airways, physicians use convex-probe endobronchial ultrasound (CP-EBUS) as a supplement to videobronchoscopy to help see target nodes situated behind airway walls. Within this context, computer-based image-guided bronchoscopy systems have seen increasing usage in the management of lung- cancer patients. In particular, our group has recently developed an integrated multimodal system for guiding both videobronchoscopy and CP-EBUS. To use this system, a guidance plan, which instructs the physician how to maneuver the device through the airways and locate target nodes for biopsy, is created off-line prior to the live procedure. The planning process, however, involves a time-consuming error prone approach requiring a skilled technician to derive. We propose an efficient automated approach for creating guidance plans for EBUS-based nodal biopsy. We also integrate this methodology into our multimodal guidance system to facilitate technician- free autonomous image-guided EBUS bronchoscopy. Patient studies show that the proposed method enables automatic reproducible computation of procedure plans without errors or dependence on technician skill. The mean planning time was reduced from 40 min per target lymph node to 5 min, with the time-consuming device maneuvers calculations reduced to a few seconds per plan. In addition, the follow-on procedure can be performed fully independent of technician assistance.

Understanding the relationship between fiducial registration error (FRE) and target registration error (TRE) is important for the correct use of interventional guidance systems. Whilst it is well established that TRE is statistically independent of FRE, system users still struggle against the intuitive assumption that a low FRE indicates a low TRE. We present the SciKit-Surgery Fiducial Registration Educational Demonstrator and describe its use. SciKit-SurgeryFRED was developed to enable remote teaching of key concepts in image registration. SciKit-SurgeryFRED also supports research into user interface design for image registration systems. SciKit-SurgeryFRED can be used to enable remote tutorials covering the statistics relevant to image guided interventions. Students are able to place fiducial markers on pre and intra-operative images and observe the effects of changes in marker geometry, marker count, and fiducial localisation error on TRE and FRE. SciKit- SurgeryFRED also calculates statistical measures for the expected values of TRE and FRE. Because many registrations can be performed quickly the students can then explore potential correlations between the different statistics. SciKit-SurgeryFRED also implements a registration based game, where participants are rewarded for complete treatment of a clinical target, whilst minimising the treatment margin. We used this game to perform a remote study on registration and simulated ablation, measuring how user performance changes depending on what error statistics are made available. The results support the assumption that knowing the exact value of target registration error leads to better treatment. Display of other statistics did not have a significant impact on the treatment performance.

Patient-to-image registration is currently mostly done in a semi-automated way. Fully automating this task can enhance surgical workflow and reduce human errors. Here, we present a novel solution with a nasal stent utilized to minimally invasively position spherical fiducials inside the nasopharynx and aimed for neurosurgery with electromagnetic navigation. The assembly was deployed into the pharyngeal region of a human specimen. The spherical fiducials are automatically detected in anatomy by integrated magnetic sensors while in preoperative imagery with a proposed u-net deep network. Temporal dislocation of the markers was measured interdaily after two brainstem procedures with CT imaging follow-ups. The u-net was trained to differentiate markers from other structures located in CT images. The ground-truth data was created from 22 CTs of phantoms, cadavers and swine. The dataset is split into 12, 5 and 5 for training, validation and testing, respectively. The dice coefficient was used as a similarity measure. The fiducial registration error resulted in 0.53±0.1 mm and 0.44±0.04 mm for the first and the second procedure, respectively. The fiducial positions deviated from the intraday baseline with the mean ± standard deviation 0.29±0.20 mm and 1.32±0.30 mm. The dice coefficient was 0.976, 0.878 and 0.764 during the training, validation and testing. The nasopharyngeal stent shows a potential for a stable marker fixation. The u-net can be adequately employed to segment titanium spherical fiducials.

During laparoscopic surgery, insufflation introduces significant deformations to the liver through reshaping the abdominal cavity. Additionally, insufflation is associated with elevation of portal and hepatic venous pressures causing hemodynamic changes that may cause swelling or engorgement of the liver. Current methods for elastic soft tissue registration cannot account for these secondary effects, which have yet remained largely neglected. This paper presents preliminary work towards modeling these intraoperative physiological changes caused by insufflation to improve the accuracy of image-to-physical registration during laparoscopic procedures. Spatially localized modes of internal pressure-driven deformations are introduced to allow reconstruction of intraoperative changes to intrahepatic pressure gradients during registration. A pilot experiment was performed in one sheep to measure volumetric tissue changes induced by insufflation. Liver volume increased by 6.3 ± 0.7% between insufflated and preoperative states, according to three independent raters. Registrations were also performed between organ states, utilizing: (1) rigid registration, (2) deformable registration based on application of external forces to the organ, and (3) the same deformable registration extended to incorporate displacements driven by changes to intrahepatic pressure gradients. Target registration error (TRE) decreased from 15.2 ± 3.2 mm with rigid registration to 5.0 ± 1.9 mm with reconstruction of external boundary forces. When incorporating internal pressure changes, TRE further decreased to 3.0 ± 1.2 mm. Additional analysis comparing a pure scaling approach with the proposed mechanics-based method to account for volume change demonstrated that the method incorporating intrahepatic pressure changes leads to more accurate image-to-physical registrations between the preoperative and insufflated liver.

PURPOSE: Point-of-care ultrasound image-guided therapies (POCUS IGT) transcend geographic and socioeconomic boundaries and help bringing modern therapies to underserved communities and countries. Unfortunately, current commercial systems are not feasible to deploy in Mauritania due to prohibitive costs of purchase, support, and operation. We present the development of a versatile POCUS IGT system, optimized for financial and operating conditions in Mauritania. We aimed to create a system that is functionally similar to the popular CIVCO product, but costs only a small fraction of the price due to the support of inexpensive ultrasound scanners and its use of open-source software. METHODS: A 3D-printed plastic needle guide with multiple guide channels was designed to securely fit around the ultrasound probe, placed in a sterile cover, and fitted with a sterile guide sleeve. Open source targeting software was developed to overlay selectable needle trajectories on the live ultrasound image. RESULTS: The assembly and workflow were found to be similar to CIVCO’s. The operator was able to accurately reach targets in the entire workspace. Excluding the price of the ultrasound scanner, the use of system costs around $1.00 per patient. CONCLUSION: A POCUS IGT system was developed using free open-source guidance software that supports low-cost ultrasound scanners, offering similar performance to the leading commercial system for 5% of price.

Ischemic heart disease remains one of the leading causes of death worldwide. Percutaneous coronary interventions (PCIs) for implanting coronary stents are preferred for patients with acute myocardial infarction but may also be performed in patients with chronic coronary syndromes to improve symptoms and outcome. During the PCI, the assessment of stent apposition, evaluation of in-stent restenosis or guidance for complex stenting of bifurcation lesions may be improved by intravascular imaging such as intravascular ultrasound (IVUS). However, advanced interpretation of the image often requires expertise and training. To approach this issue, we introduce an automatic delineation of stent struts within the IVUS pullback. We propose a cascaded segmentation based on data-driven learning with a neural encoder-decoder architecture. The learning process uses 80 IVUS sequences from 28 patients which were acquired and partially annotated by the Department of Cardiology, University Heart and Vascular Center Hamburg, Germany. The annotations include 1108, 555 and 355 frames with delineated lumen, stent and calcium as well as 13696 and 10689 frame-wise stent and no-stent indications. The network was pre-trained on lumen segmentation and refined to first identify stent frames using an encoder network and subsequently segment the struts with a decoder. Quantitative evaluation using 3-fold cross-validation revealed 88.3% precision, 92.4% recall and 0.824 Dice for the encoder and 67.0%, 60.3% and 0.611 for the decoder. We conclude that the encoder successfully leverages the larger number of high-level annotations to reject non-stent frames avoiding unnecessary false positives for the decoder trained on much less, but fine-granular annotations.

PURPOSE: High-dose-rate brachytherapy (HDR-BT) is an important treatment modality for prostate cancer that maximizes radiation dose to cancerous tissue while sparing surrounding organs. Currently, treatment planning during HDR-BT is manually completed by medical physicists, a time-consuming and observer dependent process. We propose using deep learning through a U-Net architecture to automatically segment catheters in HDR prostate brachytherapy treatment planning. METHODS: 3D Ultrasound data along with the corresponding manual contours were obtained from 49 patients undergoing HDR prostate brachytherapy. The dataset was preprocessed and then exported for training and evaluation. The resulting model was assessed both quantitatively with binary segmentation metrics and qualitatively through 3D reconstructions. RESULTS: The output segmentations demonstrated consistency on different patient datasets and good visual agreement with ground truth images. The average execution time per patient is under 30.0 s, a significant improvement from manual contouring, which may require upwards of an hour. CONCLUSION: We trained and evaluated a 3D U-Net model for automatic catheter segmentation on 3D transrectal ultrasound images generated through HDR prostate brachytherapy. Deep learning methods such as the 3D U-Net used in this scenario appear to be a promising method for automatic catheter segmentation in prostate brachytherapy.

Surgery is a crucial treatment for malignant brain tumors where gross total resection improves the prognosis. Tissue samples taken during surgery are either subject to a preliminary intraoperative histological analysis, or sent for a full pathological evaluation which can take days or weeks. Whereas a lengthy complete pathological analysis includes an array of techniques to be executed, a preliminary tissue analysis on frozen tissue is performed as quickly as possible (30-45 minutes on average) to provide fast feedback to the surgeon during the surgery. The surgeon uses the information to confirm that the resected tissue is indeed tumor and may, at least in theory, initiate repeated biopsies to help achieve gross total resection. However, due to the total turn-around time of the tissue inspection for repeated analyses, this approach may not be feasible during a single surgery. In this context, intraoperative image-guided techniques can improve the clinical workflow for tumor resection and improve outcome by aiding in the identification and removal of the malignant lesion. Hyperspectral imaging (HSI) is an optical imaging technique with the potential to extract combined spectral-spatial information. By exploiting HSI for human brain-tissue classification in 13 in-vivo hyperspectral images from 9 patients, a brain-tissue classifier is developed. The framework consists of a hybrid 3D-2D CNN-based approach and a band-selection step to enhance the capability of extracting both spectral and spatial information from the hyperspectral images. An overall accuracy of 77% was found when tumor, normal and hyper-vascularized tissue are classified, which clearly outperforms the state-of-the-art approaches (SVM, 2D-CNN). These results may open an attractive future perspective for intraoperative brain-tumor classification using HSI.

Surgery is a major treatment method for squamous cell carcinoma (SCC). During surgery, insufficient tumor margin may lead to local recurrence of cancer. Hyperspectral imaging (HSI) is a promising optical imaging technique for in vivo cancer detection and tumor margin assessment. In this study, a fully convolutional network (FCN) was implemented for tumor detection and margin assessment in hyperspectral images of SCC. The FCN was trained and tested with hyperspectral images of 25 ex vivo SCC surgical specimens from 20 different patients. The network was evaluated per patient and achieved pixel-level tissue classification with an average AUC of 0.88, 0.83 accuracy, 0.84 sensitivity and 0.70 specificity. The 95% Hausdorff distance of assessed tumor margin in 17 patients was less than 2 mm, and the classification time of each tissue specimen took less than 10 seconds. The proposed method potentially facilitates intraoperative tumor margin assessment and improves surgical outcomes.

PURPOSE: Basal Cell Carcinoma (BCC) is the most common cancer in the world. Surgery is the standard treatment and margin assessment is used to evaluate the outcome. The presence of cancerous cells at the edge of resected tissue i.e., positive margin, can negatively impact patient outcomes and increase the probability of cancer recurrence. Novel mass spectrometry technologies paired with machine learning can provide surgeons with real-time feedback about margins to eliminate the need for resurgery. To our knowledge, this is the first study to report the performance of cancer detection using Graph Convolutional Networks (GCN) on mass spectrometry data from resected BCC samples. METHODS: The dataset used in this study is a subset of an ongoing clinical data acquired by our group and annotated with the help of a trained pathologist. There is a total number of 190 spectra in this dataset, including 127 normal and 63 BCC samples. We propose single-layer and multi-layer conversion methods to represent each mass spectrum as a structured graph. The graph classifier is developed based on the deep GCN structure to distinguish between cancer and normal spectra. The results are compared with the state of the art in mass spectra analysis. RESULTS: The classification performance of GCN with multi-layer representation without any data augmentation is comparable to the previous studies that have used augmentation. CONCLUSION: The results indicate the capability of the proposed graph-based analysis of mass spectrometry data for tissue characterization or real-time margin assessment during cancer surgery.

The COVID-19 pandemic continues spreading rapidly around the world and has caused devastating outcomes towards the health of the global population. The reverse transcription-polymerase chain reaction (RT-PCR) test, as the only current gold standard for screening infected cases, yields a relatively high false positive rate and low sensitivity on asymptomatic subjects. The use of chest X-ray radiography (CXR) images coupled with deep- learning (DL) methods for image classification, represents an attractive adjunct to or replacement for RT-PCR testing. However, its usage has been widely debated over the past few months and its potential effectiveness remains unclear. A number of DL-based methods have been proposed to classify the COVID-19 cases from the normal ones, achieving satisfying high performance. However, these methods show limited performance on the multi-class classification task for COVID-19, pneumonia and normal cases, mainly due to two factors: 1) the textures in COVID-19 CXR images are extremely similar to that of pneumonia cases, and 2) there are much fewer COVID-19 cases compared to the other two classes in the public domain. To address these challenges, a novel framework is proposed to learn a deep convolutional neural network (DCNN) model for accurately classifying COVID-19 and pneumonia cases from other normal cases by the use of CXR images. In addition to training the model by use of conventional classification loss which measures classification accuracy, the proposed method innovatively employs a reconstruction loss measuring image fidelity and an adversarial loss measuring class distribution fidelity to assist in the training of the main DCNN model to extract more informative features to support multi-class classification. The experiment results on a COVID-19 dataset demonstrate the superior classification performance of the proposed method in terms of accuracy compared to other existing DL-based methods. The experiment on another cancer dataset further implies the potential of applying the proposed methods in other medical imaging applications.

PURPOSE: Identifying areas of abnormality on MRI brain scans in individuals with focal epilepsy is funda- mental to the diagnosis and treatment of the condition. However, in about a third of patients with focal epilepsy, brain scans appear to be normal (MRI-negative) as human observers cannot detect any abnormality with cur- rent imaging technology. The objective of this paper is to provide a novel approach in presenting localization using machine learning in order to locate areas of abnormality on patients with focal epilepsy on a per-voxel basis by comparing them with healthy controls. As a proof-of-concept, the technique is first applied to patients with visible lesions providing a ground truth (MRI-positive), but future work will extend this to MRI-negative subjects. METHODS: Our data consists of multi-modal brain MR images from 62 healthy control subjects and 44 MRI- positive patients with focal epilepsy. We utilized a support vector machine (SVM) as our probabilistic classifier and train it with two classes of data. We generate probability maps applying our machine learning classifier on all voxels of a test subjects to visualize the predictions. Overlap scores are used to evaluate the classifier performance in MRI-positive patients. RESULTS: Our model reached 83% specificity, 91% sensitivity, and an Area Under the Curve (AUC) of 0.896 for the task of voxel-based classification of normal versus abnormal voxels. In addition, Dice scores of up to 0.66 were achieved for the overlap measure of lesion probability map and the ground truth labels annotated by a neurologist. CONCLUSION: We demonstrated a novel approach in presenting localization using machine learning tech- niques to localize focal epilepsy lesions from multi-modal MR images.

Teleoperated robotic technology has a great potential in delivering healthcare in alternative ways from in-person encounters such as surgery, diagnosis, and nursing with specialists remotely controlled. Most teleoperated procedures heavily rely on visual feedback from cameras to observe the situation. While ideally, the camera position and orientation should be optimized adaptively depending on the task and circumstance, the view adjustment by the operator is required either by dedicating concentration on camera control or pausing during the adjustment. Then, there is the demand for a more intuitive telepresence method to improve the efficiency and performance of the remote operation. This paper proposes a hands-free approach to control the camera view for an improved telepresence experience. The system comprises an RGBD camera mounted on a robotic arm and a motion tracking virtual reality (VR) head mount display (HMD) maps the human head and upper-body motion to the robotic arm for the immersive teleoperation task. Based on this setup, an Augmented Head Motion Mapping (AHMM) mode is introduced. Wherein this mode, the user can decide to control the camera following the head motion directly or following the remote center of motion (RCM) to the target location so that the reachable visual field can be expanded. Through the user study with seven subjects, we evaluated the proposed method compared with other conventional methods in terms of the reachable visual field, control intuitiveness, and task efficiency. The possibility of further enlarging the reachable visual field by introducing the motion scaling factor is investigated through the simulation. The result successfully demonstrated that an operator using the proposed system could examine a larger area on the given object within a similar amount of time with limited training.

Purpose: As medical schools move toward competency-based medical education, they seek methods of quantifying trainee skill without human expert supervision. This study evaluates the efficacy of using object detection to track performance metrics in ultrasound-guided interventions, specifically central venous catheterization. While several studies have explored methods to automate the evaluation of these interventions, they typically rely on expensive, bulky markers. Therefore, a webcam-based approach is desirable. Methods: We used the Faster Region-Based Convolutional Neural Network object detection network developed by Ren et al. to track the two-dimensional path length and the usage time of seven tools used in central venous catheterization. Object detection relies solely on webcam imagery. Video data were collected from recordings of 20 central venous catheterization trials by four different medical students. Each recording was separated into individual frames, annotated, and inputted to the object detection network. Mean average precision was calculated for each fold and each tool. Results: The average mean average precision was 0.66. Between trials one and five, the average reduction in tool usage time was 52%, and the average reduction in 2D path length was 29%. Conclusions: The neural network was able to identify each tool with considerable accuracy. Furthermore, the neural network successfully computed differences in performance metrics that emerge as trainees gain experience. Faster Region-Based Convolutional Neural Network is an effective method to assess trainee skill in ultrasound-guided interventions

Open surgery represents a dominant proportion of procedures performed, but has lagged behind endoscopic surgery in video-based insights due to the difficulty obtaining high-quality open surgical video. Automated detection of the open surgical wound would enhance tracking and stabilization of body-worn cameras to optimize video capture for these procedures. We present results using a mask R-CNN to identify the surgical wound (the “area of interest”, AOI) in image sets derived from 27 open neck procedures (a 2310-image training/validation set and a 1163-image testing set). Bounding box application to the surgical wound was reliable (F-1 > 0.905) in the testing sets with a <5% false positive rate (recognizing non-wound areas as the AOI). Mask application to greater than 50% of the wound area also had good success (F-1 = 0.831) under parameters set for high specificity. When applied to short video clips as proof-of-principle, the model performed well both with emerging AOI (i.e., identifying the wound as incisions were developed) and with recapture of the AOI following obstruction). Overall, we identified image lighting quality and the presence of distractors (e.g., bloody sponges) as the primary sources of model errors on visual review. These data serve as a first demonstration of open surgical wound detection using first-person video footage, and sets the stage for further work in this area.

An accurate and robust image registration of computed tomography (CT) and magnetic resonance imaging (MRI) plays an important role in establishing a desired radiation treatment plan. Traditional image similarity measures such as cross-correlation, mean absolute error, mean squared error have very limited success in multi modal MRI-CT image registration. In this study, we propose a deformable registration method based on unsupervised deep neural networks to register MRI and CT for pelvic patients. No ground truth deformation vector field (DVF) is needed during training. A cross-modality image similarity loss, called as self-correlation descriptor, is used as loss function to learn the trainable parameters in deep neural networks. After training, for a new patient’s CT and MRI, the deformed MRI is obtained via first feeding the MRI and CT into the deep neural networks to derive the DVF, then deformed via spatial transformation on MRI and DVF. We evaluated our method by retrospectively revisiting 25 patients with MRI and CT acquired at pelvic region. Target registration error (TRE) was used to quantify the performance of the proposed method. The average TRE of the proposed method is 2.23±1.11 mm. It demonstrates the great potential of the proposed method in performing accurate image registration that can facilitate multimodality imaging treatment planning workflow in prostate cancer radiotherapy.

Purpose: Deep-brain stimulation via neuro-endoscopic surgery is a challenging procedure that requires accurate targeting of deep-brain structures that can undergo deformations (up to 10 mm). Conventional deformable registration methods have the potential to resolve such geometric error between preoperative MR and intraoperative CT but at the expense of long computation time. New advances in deep learning methods offer benefits to inter-modality image registration accuracy and runtime using novel similarity metrics and network architectures. Method: An unsupervised deformable registration network is reported that first generates a synthetic CT from MR using CycleGAN and then registers the synthetic CT to the intraoperative CT using an inverse-consistent registration network. Diffeomorphism of the registration is maintained using deformation exponentiation “squaring and scaling” layers. The method was trained and tested on a dataset of CT and T1-weighted MR images with randomly simulated deformations that mimic deep-brain deformation during surgery. The method was compared to a baseline method using inter-modality deep learning registration, VoxelMorph. Results: The methods were tested on 10 pairs of CT/MR images from 5 subjects. The proposed method achieved a Dice score of 0.84±0.04 for the lateral ventricles, 0.72±0.09 for the 3^rd ventricle, and 0.63±0.10 for the 4^th ventricle, with target registration error (TRE) of 0.95±0.54 mm. The proposed method showed statistically significant improvement in both Dice score and TRE in comparison to inter-modality VoxelMorph, while maintaining a fast runtime of less than 3 seconds for a typical MR-CT pair of volume images. Conclusion: The proposed unsupervised image synthesis and registration network demonstrates the capability for accurate volumetric deformable MR-CT registration with near real-time performance. The method will be further developed for application in intraoperative CT (or cone-beam CT) guided neurosurgery.

We proposed an unsupervised deep learning registration network using the combined directional local structural similarity and original images as input for multimodal CT-CBCT image registration. Due to the HU value discrepancy between CT and CBCT, conventional similarity metrics may not provide accurate image similarity measures. Directional local structural similarity measures the image’s self-similarity which reflects the underlying structural similarity regardless of the modality in use. The directional local structural similarity features could be used to drive the image deformation in conventional iterative image registration methods. A total of 30 patients’ datasets was used to train and test the network. For 15 testing cases, our results show that the alignment between the abdominal soft tissues has greatly improved after registration. The average mean-absolute error (MAE), normalized cross correlation (NCC) and target registration error (TRE) were 59.37±11.26 HU, 0.97±0.02 and 1.88±0.74 mm.

Purpose. Deep brain stimulation is a neurosurgical procedure used in treatment of a growing spectrum of movement disorders. Inaccuracies in electrode placement, however, can result in poor symptom control or adverse effects and confound variability in clinical outcomes. A deformable 3D-2D registration method is presented for high-precision 3D guidance of neuroelectrodes. Methods. The approach employs a model-based, deformable algorithm for 3D-2D image registration. Variations in lead design are captured in a parametric 3D model based on a B-spline curve. The registration is solved through iterative optimization of 16 degrees-of-freedom that maximize image similarity between the 2 acquired radiographs and simulated forward projections of the neuroelectrode model. The approach was evaluated in phantom models with respect to pertinent imaging parameters, including view selection and imaging dose. Results. The results demonstrate an accuracy of (0.2 ± 0.2) mm in 3D localization of individual electrodes. The solution was observed to be robust to changes in pertinent imaging parameters, which demonstrate accurate localization with ≥20° view separation and at 1/10th the dose of a standard fluoroscopy frame. Conclusions. The presented approach provides the means for guiding neuroelectrode placement from 2 low-dose radiographic images in a manner that accommodates potential deformations at the target anatomical site. Future work will focus on improving runtime though learning-based initialization, application in reducing reconstruction metal artifacts for 3D verification of placement, and extensive evaluation in clinical data from an IRB study underway.

Breast cancer is the most common cancer type among women. Approximately 40,000 women are expected to die from breast cancer every year. While digital mammography has a central role in the early diagnosis of breast cancer, many cancers are not visible in mammography, for example in women with dense breast tissue. Contrast enhanced magnetic resonance imaging (CE-MRI) of the breast is often used to detect lesions not visible in mammography. Lesions with suspicious characteristics on CE-MRI need to be further assessed with MRI-guided biopsy. However, MRI-guided biopsy is expensive, time consuming, and not widely available. In this paper, a novel method for a matching tool between MRI and spot mammograms is proposed. Our aim is to transfer information that is only visible in MRI onto mammographic spot projections, to enable X-ray guided biopsy even if the lesion is only visible in MRI. Two methods of registration in combination are used; a biomechanical model based registration between MRI and full view X-ray mammograms and a subsequent image based registration between full mammograms and spot mammograms. Preliminary results assessed for one patient from the Medical University of Vienna are presented. The target registration error (TRE) of biomechanical model based registration is 2.4 mm and the TRE of the image based registration is 9.5 mm. The total TRE of the two steps is 7.3 mm.

Objective: This purpose of this study was to develop and validate methods for monitoring the progression of resection during the neurosurgical removal of brain tumors. Methods: An optical tracking array was attached to a surgical instrument to track its tip position relative to preoperative imaging. The tip position was monitored continuously during surgery and used to generate a map of resection progression in the form of a 3D distance field. This method was validated on 18 brain phantoms by comparing resection maps to 3D ultrasound acquired before and after resection. A clinically compatible workflow was iteratively developed and optimized during more than 15 clinical cases with input and feedback from two neurosurgeons. Results: Phantom studies showed that resection maps accurately model the actual resection cavity, with an average of 97.5% of voxels inside the actual resection cavity lying within 2 mm of the resection map and an average of 94.7% of tracked resection points lying within 2 mm of the actual resection cavity. In clinical workflow studies, a variety of surgical instruments were tracked continuously with minimal disruption to surgery. Preliminary clinical data demonstrated that the resection map can be used to flag brain shift and predict residual tumor. Conclusion: Continuous monitoring of the progression of brain tumor resection can be done accurately and with minimal disruption in the operating room. Significance: Providing a real-time map of resection progression will reduce the cognitive load on neurosurgeons and facilitate more complete resections, particularly during long, complex surgeries.

Cochlear implants (CIs) use an array of electrodes implanted in the cochlea to directly stimulate the auditory nerve. After surgery, CI recipients undergo many programming sessions with an audiologist who adjusts CI processor settings to improve performance. However, few tools exist to help audiologists know what settings will lead to better performance. In order to provide objective information to the audiologist for programming, our group has developed a system to permit estimating which auditory neural sites are stimulated by which CI electrodes. To do this, we have proposed physics-based models to calculate the electric field in the cochlea generated by electrical stimulation. However, solving these models require days of computation time and substantial computational resources. In this paper, we proposed a deep-learningbased method to estimate the patient-specific electric fields using a U-Net-like architecture with physics-based loss function. Our network is trained with a dataset generated by solving physics-based models and the results show that the proposed method can achieve similar accuracy with traditional method and largely improves the speed of estimating the intra-cochlear electric field.

Numerous advances in CT imaging have led to improved image guidance for interventional oncology procedures. However, image artifacts stemming from metal devices used during procedures have remained a persistent problem. Severe artifacts introduced by metallic treatment devices significantly degrade image quality in anatomical regions of interest and can substantially reduce the confidence of the interventional radiologist in accurate probe placement. The purpose of this work is to develop a framework to accurately simulate the presence of metal objects in CT images, which could be used for image quality assessment and development of artifact mitigation strategies. The projection domain insertion framework was developed based on an existing lesion insertion tool. New noise and beam hardening models were developed and incorporated into the insertion algorithm to better replicate the artifactual signal produced by the metal objects. These models were validated by comparing images of real and artificially inserted metallic rods with known material composition and dimensions in head and body CT phantoms. The validated model was applied to a single cryoablation probe, which was imaged with routine clinical acquisition parameters and reconstructed with three different sets of reconstruction settings to evaluate their effects on probe segmentation and eventual insertion performance. To determine the optimal digital probe model derived from CT images, the segmented probes were inserted into the projection data from a water phantom and compared to the corresponding water images with the real probe. Additional phantom studies were conducted with two probes positioned such that they were coplanar with the imaging plane, to fully evaluate the severity of the simulated metal artifacts. Finally, a comprehensive library of metallic probes used in our clinical practice during cryoablation and microwave ablation procedures was generated, and an intuitive graphical user interface was built to facilitate efficient insertion of any number of different probes at arbitrary positions in patient CT data. Results from the metallic rod experiments demonstrated that quantum noise, electronic noise, and beam hardening were properly modeled. Further, digital probe models derived from probe images from high resolution reconstructions with an extended HU scale yielded simulated image artifacts consistent with presentation of real artifacts; whereas, using probe models derived from segmentations of clinical reconstructions or high resolution reconstruction without the extended HU scale did not. Extension of the insertion model to two coplanar probes demonstrated production of realistic artifacts with similar magnitude and texture to real probes. Lastly, the model was successfully applied to patient data and generated convincing artifacts as compared to patient images with real probes.

Purpose: A method and prototype for a fluoroscopically-guided surgical robot is reported for assisting pelvic fracture fixation. The approach extends the compatibility of existing guidance methods with C-arms that are in mainstream use (without prior geometric calibration) using an online calibration of the C-arm geometry automated via registration to patient anatomy. We report the first preclinical studies of this method in cadaver for evaluation of geometric accuracy. Methods: The robot is placed over the patient within the imaging field-of-view and radiographs are acquired as the robot rotates an attached instrument. The radiographs are then used to perform an online geometric calibration via 3D-2D image registration, which solves for the intrinsic and extrinsic parameters of the C-arm imaging system with respect to the patient. The solved projective geometry is then be used to register the robot to the patient and drive the robot to planned trajectories. This method is applied to a robotic system consisting of a drill guide instrument for guidewire placement and evaluated in experiments using a cadaver specimen. Results: Robotic drill guide alignment to trajectories defined in the cadaver pelvis were accurate within 2 mm and 1° (on average) using the calibration-free approach. Conformance of trajectories within bone corridors was confirmed in cadaver by extrapolating the aligned drill guide trajectory into the cadaver pelvis. Conclusion: This study demonstrates the accuracy of image-guided robotic positioning without prior calibration of the Carm gantry, facilitating the use of surgical robots with simpler imaging devices that cannot establish or maintain an offline calibration. Future work includes testing of the system in a clinical setting with trained orthopaedic surgeons and residents.

Precise navigation is an important task in robot-assisted and minimally invasive surgery. The need for optical markers and a lack of distinct anatomical features on skin or organs complicate tissue tracking with commercial tracking systems. Previous work has shown the feasibility of a 3D optical coherence tomography based system for this purpose. Furthermore, convolutional neural networks have been proven to precisely detect shifts between volumes. However, most experiments have been performed with phantoms or ex-vivo tissue. We introduce an experimental setup and perform measurements on perfused and non-perfused (dead) tissue of in-vivo xenograft tumors. We train 3D siamese deep learning models and evaluate the precision of the motion prediction. The network's ability to predict shifts for different motion magnitudes and also the performance for the different volume axes are compared. The root-mean-square errors are 0:12mm and 0:08mm on perfused and non-perfused tumor tissue, respectively.

The use of pedicle screws for spinal stabilization has become a common technique in spine surgery. Due to the proximity of the inserted implant to the spinal column, high accuracy is required. In the current surgical routine, the screw trajectory is planned manually on a CT scan, which is prone to error and time-consuming. We propose an automatic pedicle screw placement approach that can simplify and speed up the screw planning suitable for intra-operative use. For automatic screw positioning in the CT volume, we designed a vertebra instance patch-based approach, employing a state-of-the-art U-Net framework, which estimates a mask for the screw pair location. From the predicted screw masks, we derive the desired screw parameters. The method was trained in a 5-fold cross-validation for the lumbar spine on a large training set of 155 patients (1052 screws) and evaluated on an external test set with 30 patients (198 screws). The automatically obtained and manually defined plans for screw trajectories showed a relatively high agreement and were clinically accepted by a spinal neurosurgeon based on the Gertzbein-Robbins classification. The mean absolute difference (MAD) and corresponding standard deviation (SD) was 0.4 ± 0.3 mm for the screw diameter, 4.6 ± 3.1 mm for the screw length, 4.3 ± 2.1 mm and 4.2 ± 2.4 mm for the head and axis point and 6.0 ± 3.7° for screw insertion directions with an average inference time per vertebra of 2.9 sec. For patch initialization, we propose a manual and automatic vertebra center localization technique, facilitating the seamless integration of the method into the clinical workflow.

Instead of manual segmentation, Segmentation using Artificial Neural Networks is very useful in biomedical image analysis. However, deploying artificial neural networks requires large memory footprint and computational costs. In this work, we propose a pruning approach to alleviate these requirements for the U-Net, which is the most popular segmentation neural network for biomedical image. Our approach handles upsampling layers and skip connections, which are essential components in U-Net architecture. We show that our approach achieves 2x speedup, more than 7x size reduction with less than 2% loss in average intersection-over-union (IOU) on PhC- U373 and DIC-HeLa biomedical data set.

Parkinson's disease is a neurodegenerative disorder affecting the basal ganglia and resulting in characteristic motor and non-motor symptoms. Although pharmocological treatments are often used, deep brain stimulation can be used either to complement these treatments or replace them if ineffective. Deep brain stimulation involves the implantation of electrodes into the patient's subcortical anatomy at particular regions of interest, such as the subthalamic nucleus, in order to control or alleviate abnormal neural behaviour. For these interventions to be successful, precise pre-operative segmentation of these structures in MRI is of paramount importance. This paper presents a convolutional neural network that is capable of learning the process of subthalamic nucleus segmentation from pre-operative clinical strength MR images with an accuracy of 58:2 ± 12:1% Dice which is within the Dice range of a one-voxel translation or dilation from the reference manual segmentation. This is the final step in a combined localisation/segmentation framework for small anatomy such as the STN which is computationally efficient (avoiding deformable registration) while simultaneously being easier for the user to correct in the presence of errors.

The delineation of the prostate and organs-at-risk (OARs) is fundamental to radiation treatment planning, but is currently labor-intensive and observer-dependent. We aimed to develop an automated computed tomography (CT)-based multiorgan (bladder, prostate, rectum, left and right femoral heads) segmentation method for prostate radiation therapy treatment planning. The proposed method generated synthetic MRI (sMRI) to offer superior soft-tissue information for male pelvic CT images. Cycle-consistent adversarial networks (CycleGAN) were used to generate CT-based sMRIs. A dual attention network (DAN) extracted features from both CTs and sMRIs. A deep attention strategy was integrated into the DAN to select the most relevant features from both CTs and sMRIs to identify organ boundaries. The CT-based sMRI generated from our previously trained CycleGAN and its corresponding CT images were inputted to the proposed DAN to provide complementary information. The proposed method was trained and evaluated using datasets from 40 patients with prostate cancer, and were then compared against state-of-art methods. The Dice similarity coefficients between our results and ground truth were 0.95±0.05, 0.88±0.08, 0.90±0.04, 0.95±0.04, and 0.95±0.04 for bladder, prostate, rectum, left and right femoral heads, respectively. The proposed method could be used in routine prostate cancer radiotherapy treatment planning to rapidly segment the prostate and standard OARs.

Adaptive radiotherapy has been proposed to handle with the inter- and intra-fractional uncertainties of patient motion and anatomical changes, in which, treatment re-planning is in general a necessary procedure. Accelerated re-planning will improve the efficiency of adaptive radiotherapy, especially for those on-line adaptation. Cone beam CT (CBCT) is a widely used in-room imaging for patient positioning for modern radiotherapy. CBCT-based adaptive radiotherapy is an emerging technique for either offline or online plan adaptations. Accurately delineating tumor targets and organs-at-risk (OARs) is an important step in adaptive re-planning process, however, manual delineation can be labor-intensive and time-consuming, especially, for head and neck (HN) cancer cases, it could highly prolong the entire adaptational procedure due to the multiple OARs have to be contoured. In this study, we present a fully automated delineation method with a high quality of performance to expedite the contouring process of adaptive radiotherapy re-planning and dose-volume based plan evaluation and monitoring. In particular, a mask scoring regional neural network has been developed to extract the complementary features from CBCT and synthetic MRI for obtaining final segmentation. The synthetic MRI is generated by using a pre-trained cycle-consistent adversarial network given CBCT. Quantitative metrics including Dice similarity coefficient (DSC), Hausdorff distance 95% (HD95), mean surface distance (MSD), and residual mean square distance (RMS) were used to evaluate our proposed method. Overall, DSC values of 0.67±0.18, 0.62±0.19, 0.86±0.08, 0.87±0.13, 0.58±0.25, 0.64±0.17, 0.66±0.17, 0.93±0.05, 0.85±0.09, 0.86±0.10, 0.72±0.21, and 0.83±0.07 were achieved for brain stem, esophagus, larynx, mandible, optic chiasm, left optic nerve, right optic nerve, oral cavity, left parotid, right parotid, pharynx, and spinal cord, respectively.

Accurate segmentation of the prostate has many applications in the detection, diagnosis and treatment of prostate cancer. Automatic segmentation can be a challenging task because of the inhomogeneous intensity distributions on MR images. In this paper, we propose an automatic segmentation method for the prostate on MR images based on anatomy. We use the 3D U-Net guided by anatomy knowledge, including the location and shape prior knowledge of the prostate on MR images, to constrain the segmentation of the gland. The proposed method has been evaluated on the public dataset PROMISE2012. Experimental results show that the proposed method achieves a mean Dice similarity coefficient of 91.6% as compared to the manual segmentation. The experimental results indicate that the proposed method based on anatomy knowledge can achieve satisfactory segmentation performance for prostate MRI.

Mandibular meshes segmented from computerized tomography (CT) images contain rich information of the dentition conditions, which impairs the performance of shape completion algorithms relying on such data, but can benefit virtual planning for oral reconstructive surgeries. To locate the alveolar process and remove the dentition area, we propose a tooth segmentation method including a preprocessing step using non-rigid registration, an active contour model, and constructive solid geometry (CSG) operations. An easy-to-use interactive tool is developed, allowing users to adjust the tooth crown contour position. A validation study and a comparison study were conducted for method evaluation. In the validation study, we removed teeth for 28 models acquired from Vancouver General Hospital (VGH) and ran a shape completion test. Regarding 95th percentile Hausdorff distance (HD95), using edentulous models produced significantly better predictions of the premorbid shapes of diseased mandibles than using models with inconsistent dentition conditions(Z = -2.484, p = 0.01). The volumetric Dice score (DSC) shows no significant difference. In the second study, we compared the proposed method to manual removal in terms of manual processing time, symmetric HD95, and symmetric root mean square deviation (RMSD). The result indicates that our method reduced the manual processing time by 40% on average and approached the accuracy of manual tooth segmentation. It is promising to warrant further efforts towards clinical usage. This work forms the basis of a useful tool for coupling jaw reconstruction and restorative dentition for patient treatment planning.

The accuracy of biopsy sampling and the related tumor localization are major issues for prostate cancer diagnosis and therapy. However, the ability to navigate accurately to biopsy targets faces several difficulties coming from both transrectal ultrasound (TRUS) image guidance properties and prostate motion or deformation. To reduce inaccuracy and exam duration, the main objective of this study is to develop a real-time navigation assistance. The aim is to provide the current probe position and orientation with respect to the deformable organ and the next biopsy targets. We propose a deep learning real-time 2D/3D registration method based on Convolutional Neural Networks (CNN) to localize the current 2D US image relative to the available 3D TRUS reference volume. We experiment several scenarii combining different input data including: pair of successive 2D US images, the optical ow between them and current probe tracking information. The main novelty of our study is to consider prior navigation trajectory information by introducing previous registration result. This model is evaluated on clinical data through simulated biopsy trajectories. The results highlight significant improvement by exploiting trajectory information especially through prior registration results and probe tracking parameters. With such trajectory information, we achieve an average registration error of 2.21 mm ± 2.89. The network demonstrates efficient generalization capabilities on new patients and new trajectories, which is promising for successful continuous tracking during biopsy procedure.

We introduce a fully automatic system for cranial implant design, a common task in cranioplasty operations. The system is currently integrated in Studierfenster (http://studierfenster.tugraz.at/), an online, cloud-based medical image processing platform for medical imaging applications. Enhanced by deep learning algorithms, the system automatically restores the missing part of a skull (i.e., skull shape completion) and generates the desired implant by subtracting the defective skull from the completed skull. The generated implant can be downloaded in the STereoLithography (.stl) format directly via the browser interface of the system. The implant model can then be sent to a 3D printer for in loco implant manufacturing. Furthermore, thanks to the standard format, the user can thereafter load the model into another application for post-processing whenever necessary. Such an automatic cranial implant design system can be integrated into the clinical practice to improve the current routine for surgeries related to skull defect repair (e.g., cranioplasty). Our system, although currently intended for educational and research use only, can be seen as an application of additive manufacturing for fast, patient-specific implant design.

Due to the inter-fraction variation of anatomy, it is highly desired to provide fast and low-dose volumetric imaging during prostate radiation therapy treatment for patient setup and daily treatment dose estimation. In this study, we propose a novel generative adversarial network integrated with perceptual supervision to derive 3D volumetric images from two orthogonal 2D projections. Our proposed network, named TransNet, consists of three modules, i.e., encoding, transformation and decoding modules. Rather than only using image distance loss between the generated 3D images and the ground truth 3D CT images to supervise the network, adversarial loss is used to improve the realism of generated 3D images. We conducted a study on 20 patient cases, who had received prostate radiotherapy in our institution, and evaluated the efficacy and consistency of our method for two orthogonal projection angles, i.e., 0° and 90°. For each 3D CT image, we simulated its 2D projections at these two angles. The TransNet takes the two angles as input and output the 3D CT. The mean absolute error (MAE), peak signal-to-noise ratio (PSNR) and structural similarity index metric (SSIM) achieved by our method are 117.5±15.3HU, 22.7±3.8dB and 0.904±0.27, respectively. These results demonstrate the feasibility and efficacy of our 2D-to-3D method for prostate cancer patients, which provides a potential solution for fast on-board volumetric imaging for patient setup and adaptive radiation therapy.

Reconstructing catheters on medical images is a crucial step in high-dose-rate (HDR) brachytherapy for treating prostate cancer. However, manually identify the catheters is labor intensive. With its superior soft-tissue contrast, magnetic resonance imaging (MRI) can provide superior anatomic visualization of prostate gland and its surrounding tissues such as the rectum and the bladder. There is a considerable increase of using MRI-guided HDR prostate brachytherapy over the past decades. By incorporating MRI into prostate brachytherapy procedure, the therapeutic ratio could be improved attribute to MRI's capability of differentiating dominant prostatic lesions. However, it has been realized that challenge remains in recognition of multiple catheters in MRI because catheters used in routine HDR prostate brachytherapy appear dark, thus can be easily confused with blood vessels. In this study, we developed a deep learningbased catheter reconstruction method to tackle the challenge. Particularly, a 3D mask scoring regional convolutional neural network has been implemented to automatically identify all the catheters in MRI that are acquired after catheters insertion during HDR prostate brachytherapy. The network was trained using the paired MR images and binary catheter annotation images offered by experienced medical physicists as ground truth. After the network was trained, the locations, sizes and shapes of all the catheters can be predicted given MR images of a new prostate cancer patient receiving HDR brachytherapy. Quantities including catheter tip and shaft errors were computed to assess our proposed method. Our method detected 164 catheters from 11 patients receiving HDR prostate brachytherapy with a catheter tip error of 0.62±1.83 mm and a catheter shaft error of 0.94±0.52 mm. The proposed multi-catheter reconstruction method has the capability of precisely localizing the tips and shafts of catheters in 3D MRI images of HDR prostate brachytherapy. It paves the way for elevating the quality and treatment outcome of MRI-guided HDR prostate brachytherapy.

Delineation of organs-at-risk (OARs) is a labor-intensive and time-consuming procedure in radiation treatment planning process. This work aims to develop a deep learning-based method for rapid and accurate pancreatic multi-organ delineation with the goal of expediting the treatment planning process. A retrospective investigation has been carried out on twenty patients with contours delineated on CT simulation scan. Eight OARs including large bowel, small bowel, duodenum, left kidney, right kidney, liver, spinal cord and stomach were selected for automated segmentation. In order to achieve expert level of accuracy, we proposed a regional proposal network which consists of two stages of feature extraction from given CT images. A coarse feature map was first obtained using a backbone network, and then the refined feature map was extracted using mask regional convolutional neural network. Metrics including Dice similarity coefficient (DSC), sensitivity, specificity, Hausdorff distance 95% (HD95), mean surface distance (MSD) and residual mean square distance (RMSD) were computed to quantitatively assess the performance of our proposed method. DSC values of 0.90±0.09, 0.87±0.13, 0.82±0.12, 0.95±0.03, 0.95±0.06, 0.96±0.02, 0.90±0.03, and 0.94±0.02 were achieved for large bowel, small bowel, duodenum, left kidney, right kidney, liver, spinal cord and stomach, respectively.

Intravascular optical coherence tomography (IVOCT) images have the necessary resolution and contrast to assess atherosclerotic plaque, and our group has created machine/deep learning techniques to analyze plaques in pre-stent IVOCT images. Such software can be used to aid numerous research studies and clinical treatment planning. Here we extend our work to images obtained following stent implantation, where opaque metallic stent struts obstruct the view of portions of the plaque, making it difficult to visualize/analyze plaques. We created a generative adversarial network (GAN)-based method to fill in the missing image data. Applications include analysis of plaque progression/regression behind the stent to evaluate drugs, drug eluting stents, and the anatomical registration of pre- and post-stent image data for further post stent optimization. We used conditional GAN (cGAN) to correct the presence of the guide wire and stent struts shadows, creating full images for visualization and analyses. To train/test software, we created synthetic post-stent images by interjecting realistic stent strut shadows in images without a stent in place. Results show the capability of cGAN to generate plausible and realistic images. To assess results, we used deep learning segmentation models to segment calcifications in corrected, synthesized images and compared results to the original images with stent. DICE scores were typically above 0.79 ± 0.03 with correction and 0.71 ± 0.02 without correction. The co-registration errors improved to be ranged between 0.52 mm to 0.69 mm. Compared with images with shadows, the new method offers lower errors in both location and orientation registration.

With lung cancer being the most fatal cancer worldwide, it is important to detect the disease early. A potentially effective way of detecting early cancer lesions developing along the airway walls (epithelium) is bronchoscopy. To this end, developments in bronchoscopy offer three promising noninvasive modalities for imaging bronchial lesions: white-light bronchoscopy (WLB), auto uorescence bronchoscopy (AFB), and narrow-band imaging (NBI). While these modalities give complementary views of the airway epithelium, the physician must manually inspect each video stream produced by a given modality to locate the suspect cancer lesions. Unfortunately, no effort has been made to rectify this situation by providing efficient quantitative and visual tools for analyzing these video streams. This makes the lesion search process extremely time-consuming and error-prone, thereby making it impractical to utilize these rich data sources effectively. We propose a framework for synchronizing multiple bronchoscopic videos to enable interactive multimodal analysis of bronchial lesions. Our methods first register the video streams to a reference 3D chest computed-tomography (CT) scan to produce multimodal linkages to the airway tree. Our methods then temporally correlate the videos to one another to enable synchronous visualization of the resulting multimodal data set. Pictorial and quantitative results illustrate the potential of the methods.

Intraoperative stereovision (iSV) systems are used for image data acquisition of the surgical field to facilitate image updating in image-guided surgery. We have developed an optically-tracked hand-held stereovision (HHS) system, but the calibration process was lengthy and the accuracy can be greatly affected by localization errors. This study shows the efficacy of using a rigid calibration checkerboard contained within a custom-built tracking frame for semi-automated iSV calibration. First, stereo camera parameters were calculated with a mean reprojection error of 0.28 pixels using 46 image pairs. Then, the checkerboard was calibrated to optically track all corner points. Specifically, locations of 4 outermost corner points on the checkerboard pattern were collected 93 times using a tracked stylus, and internal corner points were interpolated. Computed spacing between corner points was 15.01±0.04 mm and 15.05±0.08 mm in x and y directions, respectively, compared to the advertised spacing of 15 mm for both x and y directions. Next, we captured 169 image pairs of the tracked calibration checkerboard to reconstruct the 3D coordinates of corner points in the camera space. After, we calculated a rigid transformation between the reconstructed points and their tracked locations with a mean error of 0.31±0.02 mm. Finally, another 50 image pairs of the tracked checkerboard were used to verify the calibration accuracy and showed a mean error of 0.88±0.24 mm. These results show promise to automate the calibration process for future units and settings.

Endoscopic video sequences provide surgeons with much structural information (e.g., vessels and neurovascular bundles) that guides them to accurately manipulate various surgical tools and avoid surgical risks. Unfortunately, it is difficult for surgeons to intuitively perceive these small structures with tiny pulsation motion on endoscopic images. This work proposes a new endoscopic video motion magnification method to accurately generate the amplified pulsation motion that can be intuitively and easily visualized by surgeons. The proposed method explores a new temporal filtering for Eulerian motion magnification method to precisely magnify the tiny pulsation motion and simultaneously suppress noise and artifacts in endoscopic videos. We evaluate our approach on surgical endoscopic videos acquired in robotic prostatectomy. The experimental results demonstrate that our proposed temporal filtering method essentially outperforms other filters in current video motion magnification approaches, while it provides better visual quality and quantitative assessment than other methods.

Endoscopic illumination system designs are inherently space restricted resulting in non-uniform illumination patterns. Tubular tissues and organs have significant lengths relative to these fixed illumination patterns, resulting in dark areas further away from the imaging sensor and over-saturated areas closer to the sensor. Occlusions and obstructions in the illumination path can also result in significant shadowing. A gamma correction is normally applied to map luminance levels and adjust for the significant variations that can occur. During the process, the color information isn’t taken into consideration. In this paper, a new gamma correction method was proposed to increase image contrast and sharpness by utilizing pixel information from all color channels. Raw data was collected for various scenes and processed using the regular and proposed gamma correction method. Images were quantitatively compared using entropy and sharpness metrics. Both subjective interpretation and these statistics show that the proposed gamma correction method offers enhancement over the regular gamma correction method by increasing the entropy and maintaining the sharpness closer to the original image.

Office-based endoscopic laser surgery is an increasingly popular option for the treatment of many benign and premalignant tumors of the vocal folds. While these procedures have been shown to be generally safe and effective, recent clinical studies have revealed that there are a number of challenging locations inside the larynx where laser light cannot be easily delivered due to line-of-sight limitations. In this paper, we explore whether these challenges can be overcome through the use of side-firing laser fibers. Our study is conducted in simulation, using three-dimensional models of the human larynx generated from X-ray microtomography scans. Using computer graphics techniques (ray-casting), we simulate the application of laser pulses with different types of laser fibers and compare the total anatomical coverage attained by each fiber. We consider four fiber types: a traditional “forward-looking” fiber - not unlike the ones currently used in clinical practice - and three side-firing fibers that emit light at an angle of 45, 70, and 90 degrees, respectively. Results show that side-firing fibers enable a ∼70% increase in accessible anatomy compared to forward-looking fibers.

Electroporation is a technique that uses short-duration (nanosecond to microsecond) high-voltage electric pulses to create temporary pores in cell membranes or irreversibly destroys target cells for cancer treatment. The versatility of electroporation makes the technique ideal for direct drug delivery and cancer treatment. Electroacoustic tomography (EAT) is a novel imaging modality for electroporation treatment monitoring, that utilizes the existing energy deposited by highvoltage electrical fields created in the electroporation process. In EAT, ultrasound waves are generated by the deposited electric potential energy and captured by different configurations of ultrasound transducers. After being captured, the ultrasound information is used to reconstruct the geometric information of the electric field distribution in the target medium. In this work, we demonstrate the feasibility of electroacoustic tomography (EAT) by creating a 2D reconstruction of an electric field in the water. A custom high-voltage electric pulse generating device was made for delivering nanosecond duration pulses in water through two electrodes. A 500Khz ultrasound transducer was placed near the electrodes and rotated to capture the acoustic information from 180 positions. The corresponding 2D reconstruction of the electric field in water was completed by using filtered back-projection algorithm. The successful reconstruction of the electrical field in water marks the important step of EAT development for in-situ electroporation monitoring.

Treatment planning is an essential step in radiation therapy (RT). It requires solving a complex inverse optimization problem through time-consuming adjustment of a set of parameters to meet clinical objectives in a trial and error fashion. Fast and robust treatment plan optimization is important to achieve effective RT of cancer patients. We propose an artificial intelligence (AI)-based RT planning strategy that uses a deep-Q reinforcement learning (RL) to automatically optimize machine parameters by finding an optimal machine control policy. The network uses RT planning CT and contours as input, and dual deep-Q networks that are trained to control the dose rate and multi-leaf collimator positions based on the current dose distribution and machine parameter state. The Q-value is computed as the discounted cumulative cost based on dose objectives, and minimized by experience-replay RL to determine the policy. The proposed approach was applied to prostate cancer RT planning, and validated on 10 prostate cancer cases. Dose distributions generated by RL were compared to conformal arcs and clinical intensity modulated radiotherapy (IMRT) plans. RL was able to generate RT plan with comparable target and normal tissue dose to clinical plans with mean±SD doses of 83.1±1.7 Gy, 39.9±10.0 Gy, and 39.6±13.9 Gy for planning target volume (PTV), rectum, and bladder, respectively (vs 84.4±1.0 Gy, 41.8±15.0 Gy, and 50.6±11.4 Gy for clinical IMRT). This preliminary study demonstrates the potential of an RL approach to enable rapid AI-based RT plan optimization, significantly reducing time and burden required for RT planning.

In this work, we compare five deep learning solutions to automatically segment the resection cavity in postoperative MRI. The proposed methods are based on the same 3D U-Net architecture. We use a dataset of postoperative MRI volumes, each including four MRI sequences and the ground truth of the corresponding resection cavity. Four solutions are trained with a different MRI sequence. Besides, a method designed with all the available sequences is also presented. Our experiments show that the method trained only with the T1 weighted contrast-enhanced MRI sequence achieves the best results, with a median DICE index of 0.81.

In this study, we propose a novel deep learning-based method to predict dose distribution based on patient’s pancreas planning CT and organ contours. To comprehensively extract features from CT and contours, dual pyramid networks (DPNs) with late fusion network (LFN) are used. The proposed network consists of three subnetworks, i.e., CT-only feature pyramid network (FPN), contour-only feature pyramid network (FPN) and late fusion network (LFN). CT-only FPN and contour-only FPN are proposed to learn complementary tissue-contrast information and semantic information from CT and contour, respectively. LFN is proposed to combine the pyramid features from CT and contour, and finally perform dose prediction. To select the most relevant features that well-represent the dose distribution, a deep attention strategy was integrated into the LFN. A retrospective study on 30 patients’ pancreas CT and contour was used to evaluate the proposed method’s efficiency. The proposed DL model was able to predict dose metrics for PTV (p = 0.93), stomach (p = 0.91) and duodenum (p = 0.27) without significant differences compared to the ground truth. The model, however, overestimated dose to the 50% of liver volume. Overall results demonstrate the feasibility and efficacy of our deep learning-based method for pancreas SBRT dose prediction.

Cochlear implants (CIs) are neuroprosthetic devices that can improve hearing in patients with severe-to-profound hearing loss. Postoperatively, a CI device needs to be programmed by an audiologist to determine parameter settings that lead to the best outcomes. Our group has developed an image-guided cochlear implant programming (IGCIP) system to simplify this laborious post-programming procedure and improve hearing outcomes. IGCIP requires image processing techniques to analyze the location of the inserted electrode arrays (EAs) with respect to the intracochlear anatomy (ICA). An active shape model (ASM)-based method is currently in routine use in our IGCIP system for ICA segmentation. Recently, we have proposed a hybrid ASM/deep learning (DL) segmentation method that improves segmentation accuracy. In this work, we first evaluate the effect of this method on so-called distance-vs.-frequency curves (DVFs), which permit to visualize electrode interaction and are used to provide programming guidance. An expert evaluation study is then performed to manually configure the electrodes based on the DVFs and grade the quality of the electrode configurations derived from ASM and hybrid ASM/DL segmentations compared to the one derived from ground truth segmentation. Results we have obtained show that the hybrid ASM/DL segmentation technique tends to generate DVFs with smaller frequency error and distance error, and electrode configurations which are comparable to the existing ASM-based method.

Purpose: The objective of this paper is to present heatmaps from the likelihood of clinically significant prostate cancer with a deep learning model. This will give radiologists more information on the location of prostate lesions. Methods: 3D Slicer module was developed using a machine learning model to predict pixel-by-pixel PI-RADS scores. The working hypothesis is that the machine learning algorithm will be capable of producing heatmaps with hotspots within a typical size of a lesion with PI-RAD score of 4. Discussion and conclusion: The study provided insight into the future of MRI assessment using Deep Learning models.

Purpose. Surgical placement of pelvic instrumentation is challenged by complex anatomy and narrow bone corridors, and despite heavy reliance on intraoperative fluoroscopy, trauma surgery lacks a reliable solution for 3D surgical navigation that is compatible with steep workflow requirements. We report a method that uses routinely acquired fluoroscopic images in standard workflow to automatically detect and localize orthopaedic instruments for 3D guidance. Methods. The proposed method detects, establishes correspondence of, and localizes orthopaedic devices from a pair of radiographs. Instrument detection uses Mask R-CNN for segmentation and keypoint detection, trained on 4000 cadaveric pelvic radiographs with simulated guidewires. Keypoints on individual images are corresponded using prior calibration of the imaging system to backproject, identify, and rank-order ray intersections. Estimation of 3D instrument tip and direction was evaluated on a cadaveric specimen and patient images from an IRB-approved clinical study. Results. The detection network successfully generalized to cadaver and clinical images, achieving 87% recall and 98% precision. Mean geometric accuracy in estimating instrument tip and direction was (1.9 ± 1.6) mm and (1.8 ± 1.3)°, respectively. Simulation studies demonstrated 1.1 mm median error in 3D tip and 2.3° in 3D direction estimation. Preliminary tests in cadaver and clinical images show the basic feasibility of the overall approach. Conclusions. Experimental studies demonstrate the feasibility and highlight the potential of deep learning for 3D-2D registration of orthopaedic instruments as applied in fixation of pelvic fractures. The approach is compatible with routine orthopaedic workflow, does not require additional equipment (such as surgical trackers), uses common imaging equipment (mobile C-arm fluoroscopy), and does not require vendor-specific device models.

Percutaneous techniques are becoming more widely adopted for treating solid organ tumours. With many of these techniques using image guidance, accuracy is essential to reduce the risk of complications and tumour recurrence. We propose a novel approach to needle trajectory forecasting using a 3D U-Net trained on interventional computed tomography (iCT) images from renal cryoablation procedures. The U-Net is trained to predict future needle locations from present iCT images, supervised by ground truth labels at future time points. Furthermore, we demonstrate that forecasting needle trajectory may be substantially improved by Monte Carlo dropout (MCDO), in addition to generating uncertainty maps of the predictions. With 122 training iCT volumes, a Dice coefficient of 0.48 on highly elongated needle morphology was achieved based on 41 unseen test cases, significantly outperforming two models not using MCDO at inference with Dice scores of 0.14 and 0.32. MCDO also greatly improved the predicted needle morphology and was able to incorporate directional information into the predictions. Our approach shows promise for improving accuracy and workflow in image-guided procedures, with an interesting research direction to predict uncertain future interventional events by ensemble in supervised approaches.

The rise in number of fluoroscopy-guided procedures has prompted strategies for radiation management in order to minimize exposure to ionizing radiation in patients and healthcare personnel. The FluoroShield system utilizes an artificial intelligence (AI) approach towards radiation exposure reduction. The system automatically tracks the region of interest (ROI) in real-time, controls a rapid lead shutter for collimating the X-ray beam to the ROI, and blends the imaged ROI with the entire field of view that is updated at a lower frame rate to present, at all times, a full image to the operator. In this work, we discuss the AI-based component of the system that is used for tracking the ROI in endoscopic procedures. Known as the auto-ROI processor (ARP), the methodology comprises a merged architecture of convolutional neural networks for object detection that allows for both fine and contextual features to be captured. The detection probabilities are incorporated within a particle filtering framework for spatial and temporal updating of the ROI in a Bayesian approach. The ARP's performance is evaluated on fluoroscopic sequences, taking into consideration the estimated radiation reduction rates. The reduction in radiation exposure based on simulations is comparable to the reported values in a clinical study.

A deep-learning model based on the U-Net architecture was developed to segment multiple needles in the 3D transrectal ultrasound (TRUS) images. Attention gates were adopted in our model to improve the prediction on the small needle points. Furthermore, the spatial continuity of needles was encoded into our model with total variation (TV) regularization. The combined network was trained on 3D TRUS patches with the deep supervision strategy, where the binary needle annotation images from simulation CTs were provided as ground truth. The trained network was then used to localize and segment the HDR needles for a new patient TRUS images during high-dose-rate (HDR) prostate brachytherapy. The needle shaft and tip errors against CT-based ground truth were used to evaluate other methods and other methods as comparison. Our method detected 96% needles of 339 needles from 23 HDR prostate brachytherapy patients with 0.29±0.24 mm at shaft error and 0.442±0.831 mm at tip error. For shaft localization, our method resulted in 96% localizations with less than 0.8 mm error (needle diameter is 1.67 mm), while for tip localization, our method resulted in 75% needles with 0 mm error and 21% needles with 2 mm error (TRUS image slice thickness is 2 mm). No significant difference was observed (p = 0.83) on tip localization between our results with the ground truth. Compared with U-Net and deep supervised attention U-Net, the proposed method delivers a significant improvement on both shaft error and tip error. Besides, to our best knowledge, this is the first attempt on multi-needle localization in the prostate brachytherapy. The 3D rendering of the needles could help clinicians to evaluate the needle placements. It paves the way for the development of real-time radiation plan dose assessment tools that can further elevate the quality and outcome of prostate HDR brachytherapy.

Fusion of pre-operative Magnetic Resonance Imaging (MRI) and Trans-Rectal Ultrasound (TRUS) guided biopsy (Fusion Biopsy) has proven to be more effective as compared to cognitive biopsy for the detection of prostate cancer. The detection of the biopsy needle used during the Ultrasound procedure has multiple applications like reporting, repeat biopsy planning and planning therapy. Earlier methods to solve this problem have only used image processing techniques like Hough- Transform or Graph-Cut. These techniques lack robustness because only image-based solution cannot take care of the huge variability in the data as well as the problem of needle going out of plane. Recent deep learning (DL) based solutions for needle detection have high latency and does not exploit temporal information present in TRUS imaging. In this paper, we propose a method to automatically detect the short-lived needle triggers and its position using temporal context incorporated into a DL model termed as Samsung Multi-Decoder Network (S-MDNet). The proposed solution has been tested on 8 patients and yields high sensitivity (96%) and specificity (95%) for the detection of the needle trigger event.

MR-guided high-intensity focused ultrasound (MR-HIFU) is a non-invasive therapeutic technology which has demonstrated clinical potential for tissue ablation. The application of this therapeutic approach facilitated to be a promising option to achieve faster pain palliation in patients with bone metastasis. However, its clinical adoption is still hampered by a lack of workflow integration. Currently, to ensure sufficient positioning, MR images have to be repeatedly acquired in between patient re-positioning tasks, leading to a time-consuming preparation phase of at least 30 minutes, involving extra costs and time to the available treatment time. Augmented reality (AR) is a promising technology that enables the fusion of medical images, such as MRI, with the view of an external camera. AR represents a valid tool for a faster localization and visualization of the lesion during positioning. The aim of this work is the implementation of a novel AR setup for accelerating the patient positioning during MRHIFU treatments by enabling adequate target positioning outside the MRI scanner. A marker-based approach was investigated for fusing the MR data with video data for providing an augmented view. Initial experiments on four volunteers show that MR images were overlaid on the camera views with an average re-projection error of 3.13 mm, which matches the clinical requirements for this specific application. It can be concluded that the implemented system is suitable for MR-HIFU procedures and supports its clinical adoption by improving the patient positioning, thereby offering potential for faster treatment time.

Cardiac catheterization is a delicate strategy often used during various heart procedures. However, the procedure carries a myriad of risks associated with it, including damage to the vessel or heart itself, blood clots, and arrhythmias. Many of these risks increase in probability as the length of the operation increases, creating a demand for a more accurate procedure while reducing the overall time required. To this end, we developed an adaptable virtual reality simulation and visualization method to provide essential information to the physician ahead of time with the goal of reducing potential risks, decreasing operation time, and improving the accuracy of cardiac catheterization procedures. We additionally conducted a phantom study to evaluate the impact of using our virtual reality system prior to a procedure

Brachytherapy is often used in gynecologic cancer treatment to provide high radiation doses to tumors and spare nearby healthy tissues. Intracavitary applicators, including tandem-and-ovoid and tandem-and-ring, are commonly used to position the radioactive sources appropriately. Three-dimensional (3D) transrectal ultrasound (TRUS) imaging has been demonstrated to allow for consistent delineation of the clinical target volume; however, the ability to visualize applicators and relevant structures following applicator insertion has not been investigated. We propose the use of a 3D TRUS system to visualize applicators at the time of placement. In two patient images, the key components of the tandem-and-ovoid applicators were clearly visualized, as well as the uterus, cervix, and vagina, with the potential to identify the tumor and organs-at-risk in these images. Although the tandem-and-ring applicator (one patient) obscured the cervix and anterior anatomy, the posterior applicator edges were visualized and we propose combining the 3D TRUS image with a 3D transabdominal ultrasound (TAUS) image for more complete visualization of the necessary structures. We designed a multimodality application-specific pelvic phantom to assess the feasibility of the image fusion and performed preliminary feasibility assessment on a tandem-and-ovoids applicator and a tandem-and-ring applicator with an interstitial ring cap. The resulting phantom images showed promising features for future image fusion. Intraoperative assessment of applicator placement has the potential to improve the treatment quality and reduce the risk of complications from overexposure of nearby normal tissues, as well as provides a promising approach for accessible image-guided brachytherapy, facilitating broader adoption to healthcare cost-constrained settings.

The simulation of realistic ultrasound (US) images has many applications in image-guided surgery such as image registration, data augmentation, or education. We simulated intraoperative US images of the brain after tumor resection surgery. A Generative Adversarial Networks first generated an US image with resection from a resection cavity map. This generated cavity texture was then blended into a real pre-resection patient-specific US image. A validation study showed that two neurosurgeons correctly labelled only 56% and 53% of the simulated images, which indicate that these synthesized images are hardly distinguishable from real post-resection US images.

Cochlear implants consist of an implanted array of electrodes and passive electronics and an external processor designed to directly stimulate auditory nerve fibers to induce the sensation of hearing in those who have experienced hearing loss caused by problems in the inner ear. After surgical implantation, audiologists program the processor with settings intended to produce optimal hearing outcomes. The likelihood of optimal outcomes increases when audiologists are provided with tools that assist them in making objective decisions based on the patient’s own anatomy and surgical placement of the array. A visualization tool currently in use, called distance-versus-frequency (DVF)curves, can be used to estimate channel interactions between electrodes. Although the information presented is objective, an audiologist’s decisions are subject to their own estimations applied to this visualization. In this paper, we present a new visualization technique designed to remove the subjectivity of visually assessing channel interactions between electrodes. Preliminary results show plans generated using this method are more consistent and are rated as optimal more often than those generated using DVF curves.

While mainly used for reducing seizures in epileptic patients, vagus nerve stimulation (VNS) has been implied to be capable of treating various other diseases. However, the therapeutic extent and control of neuromodulation for these conditions are still uncertain and are limited by the ability to predict neural activation responses upon targeting certain fascicles. Generally, VNS is administered through a bipolar helical cuff electrode implanted around the left vagus nerve. The electrode delivers pulses of electricity to the nerve to recruit axons. This work focuses on predicting percent activation and regions of activation based on different adjustable factors such as injected current amplitude and pulse width of stimuli and the activated region of the electrode. To achieve, a simplified finite element model was created using cylindrical geometries as nerve components with an addition of helical cuff electrodes. All of these components were encased in surrounding tissue with assumed properties similar to adipose. Electric potential distribution in the model was processed with an activating function defined along the axonal length which estimates the injected current at the nodes of Ranvier (NoR). These values were then enforced in the neuron simulation as a current clamp approximation applied at the NoR and the likelihood of an action potential was determined. Presence of action potentials were then detected to determine which axons were recruited in VNS. This preliminary work determined that electrode configurations can target specific fascicles in the nerve while amplitudes and pulse widths of stimuli contribute to the percentage of the nerve activated. This demonstrates the ability for patient-specific control over targeting fascicles. Additionally, this work presents initial steps to improving the model by using histological data to create a geometric-specific approach.

Brain shift is a confounder to the accuracy of electrode lead placement during deep brain stimulation (DBS) surgery. Model based image updating method can compensate for brain shift with high efficiency and accuracy. A key element to achieving clinically accepted accuracy using our biomechanical brain model is designating rigorous boundary conditions (BCs) that define general physics of the model. In this retrospective study, we searched for a set of six optimal BCs such as gravitational direction and level of CSF for our model to simulate accurate brain shift in DBS lead placement surgery. Specifically, we conducted 9072 trials of brain shift simulation with varying boundary conditions and deep brain sparse data for three training cases and applied these parameters to three testing cases for evaluation. The optimal set of parameters was determined based on lowest target registration error (TRE) evaluated at five deep brain landmarks near the subthalamus area. We show that simulations with optimal BCs compensated 61.28% and 50.06% of brain shift on average in two of the three testing cases where large brain deformation occurred and 26.5% in one testing case of small brain shift. In comparison, optimal BCs delivered consistent and accurate prediction of brain shift at all deep brain landmarks in both training and testing cases whereas default sets of BCs produced similar results at some landmarks but underperformed for the rest. With only deep brain sparse data and a set of optimal BCs, our biomechanical brain model can achieve significant brain shift compensation in DBS cases and Its clinical utility will be examined in surgical cases in future OR.

Accurate classification of pulmonary nodules in the CT images is critical for early detection of lung cancer as well as the assessment of the effect from COVID-19. In this paper, we propose a computer-aided classification method for lung nodules using expert knowledge. We use a decoupling metric learning model to describe the deep characteristics of the nodules and then calculate the similarity between the current nodule and the nodules in the database. By analyzing the returned nodules with the diagnosis information, we obtain the expert knowledge of similar nodules, based on which we make the decision of the current nodule. The proposed method has been evaluated on the benchmark LIDC-IDRI dataset and achieved an accuracy of 95.7% and AUC of 0.9901. The proposed classification method can have a variety of applications in lung cancer detection, diagnosis and therapy.

PURPOSE: Prostate cancer is the second most common cancer diagnosed in men. The rate is disproportionately high among men in sub-Saharan Africa where, unlike in North America and Western Europe, the screening process for prostate cancer has historically not been routine. Currently, as awareness regarding prostate health increases, more patients in this region are being referred to trans-rectal ultrasound guided prostate biopsy, a diagnosis procedure which requires a strong understanding of prostate zonal anatomy. To aid in the instruction of this procedure, prostate biopsy training programs need to be implemented. Unfortunately, current TRUS-guided training tools are not ideal for reproducibility in these Western African countries. To answer this challenge, we are developing an affordable and open-source training simulator for TRUS-guided prostate biopsy, for use in Senegal. In this paper, we present the implementation of the training simulator’s virtual interface, highlighting the generation and evaluation of the critical training component of zonal anatomy overlaid on TRUS. METHODS: For the simulator’s dataset, we registered TRUS and MRI volumes together to obtain the zonal segmentation from the MRI volumes. After generating ten pairings of TRUS overlaid with zonal segmentation, we designed and implemented a virtual TRUS training system, developed in open-source software. The objective of our simulator is to teach trainees to accurately identify the prostate’s anatomical zones in TRUS. To confirm the system’s usability for training zonal identification, we conducted a two-part survey on the quality of the zonal overlays with 7 urology experts. In the first part, they assessed the zonal overlay for visual correctness by rating 10 images from one patient’s TRUS with registered overlay on a 5-point Likert scale. For the second part, they labelled 10 plain TRUS volumes with zonal anatomy and the labels were compared to the labels of our overlay. RESULTS: On average, experts rated the zonal overlay’s visual accuracy at 4 out of 5. Furthermore, 7 out of 7 experts labelled the peripheral, anterior, and transitional zones in the same regions we overlaid them, and 5 out of 7 labelled the central zone in the same region we overlaid it. CONCLUSION: We created the prototype of a TRUS imaging simulator in open-source software. A vital training component, zonal overlay, was generated using publicly accessible data and validated by expert urologists for prostate zone identification, confirming the concept.

Publisher’s Note: This paper, originally published on 15 February 2021, was replaced with a corrected/revised version on 13 May 2021. If you downloaded the original PDF but are unable to access the revision, please contact SPIE Digital Library Customer Service for assistance.