By: Vaibhavi Sahane, RIG Intern Researcher


One of the most promising areas of health innovation is the application of artificial intelligence (AI), primarily in medical imaging. AI is the essential boosting power of processing massive numbers of medical images and potentially uncovering disease characteristics that fail to be identified by humans alone. Researchers have applied AI to automatically recognize complex patterns in imaging data and provide quantitative assessments of radiographic characteristics. In radiation oncology, AI has been applied on different image modalities that are used to support different stages of treatment. i.e., tumor delineation and treatment assessment. This blog discusses the role of machine learning, deep learning and neural networks for processing medical imaging, challenges for AI in medical imaging research and clinical applications.



Machine Learning (ML) incorporates computational models and algorithms that imitate the architecture of the biological neural networks in the brain, i.e., artificial neural networks (ANNs). Neural network architecture is structured in layers composed of interconnected nodes. Each node of the network performs a weighted sum of the input data that is subsequently passed to an activation function. There are three different kinds of layers: the input layer, which receives input data; the output layer, which produces the results of data processing; and the hidden layer(s), which extracts the patterns within the data. The Deep Learning (DL) approach was developed to improve on the performance of conventional ANNs when using deep architectures. Among the different deep ANNs, convolutional neural networks (CNNs) have become popular in computer vision applications. “In this class of deep ANNs, convolution operations are used to obtain feature maps in which the intensities of each pixel/voxel are calculated as the sum of each pixel/voxel of the original image and its neighbors, weighted by convolution matrices (also called kernels).” Different kernels are applied for specific tasks, such as blurring, sharpening, or edge detection. “CNNs are biologically inspired networks mimicking the behavior of the brain cortex, which contains a complex structure of cells sensitive to small regions of the visual field.” The architecture of deep CNNs allows for the composition of complex features (such as shapes) from simpler features (e.g., image intensities) to decode image raw data without the need to detect specific features.

Fig 1: Comparison between classic machine learning and deep learning approaches applied to a classification task. Both depicted approaches use an artificial neural network organized in different layers (IL input layer, HL hidden layer, OL output layer). The deep learning approach avoids the design of dedicated feature extractors by using a deep neural network that represents complex features as a composition of simpler ones. ** Image referred from article: Artificial intelligence in medical imaging: threat or opportunity? Radiologists again at the forefront of innovation in medicine


Despite their performance, ML network architectures make processing image data more prone to fail in reaching the convergence and overfit of the training dataset.

Success in DL application was possible due to recent advancements in the development of hardware technologies, such as graphics processing units. Indeed, the high number of nodes needed to detect complex relationships and patterns within data may result in billions of parameters that need to be optimized during the training phase. For this reason, DL networks require a huge amount of training data, which in turn increases the computing power needed to analyze them. These are also the reasons why DL algorithms are showing increased performance and are, theoretically, not susceptible to the performance plateau of the simpler ML networks.


Two challenges need to be resolved before AI can be more widely implemented in medical imaging research: first, is how to organize and pre-process data generated from different institutions. Many authors have successfully derived patient representations from large-scale data sets that were not optimized for any specific task and can fit different clinical applications. However, their data is from one institution. Tackling data sets from multiple institutions is a much more challenging task because even for the same procedure, different institutions might implement differently. Patient cohorts might also be different. All of these concerns will need to be addressed when pre-processing the input data for an AI algorithm.

The second challenge, is on a policy or infrastructure level and how institutions can be encouraged to share more image data. Currently, image data sharing is very limited. One major concern is the lack of infrastructure to house medical data securely which needs to work in parallel with the emerging needs of data sharing.


Cancer Diagnosis and Characterization: AI technologies are often employed to help with the detection of cancer which can potentially reduce healthcare costs due to misdiagnosis and aid in the transition towards novel precision medical protocols. AI can also be used to characterize cancer by describing tumor gene mutation status or infiltration of nearby structures. Microcalcifications, which can potentially be an early sign of breast cancer, can be detected from mammograms using ML clustering methods such as k-means or DL, which also allows segmentation of microcalcifications and breast parenchyma (breast parenchyma is essentially breast density in comparison of the relative amounts of fat versus fibroglandular tissues in the breast.) Models for discriminating malignant lesions in digital mammograms, which make use of RF binary classifiers have been proposed. The development of deep CNN (DCNN) models have been shown to improve the diagnostic accuracy in discriminating malignant breast cancer lesions in mammography. A DCNN architecture was proposed for reducing false negatives while still keeping acceptable accuracy, showing that random initialization CNN architecture can provide practical aid in the classification and staging of breast cancer. Recently, AI was applied to a novel instrumentation for diagnosis of breast cancer called Contrast-enhanced spectral mammography (CESM), where dual-energy mammograms are acquired after contrast medium administration. It can also provide images where only the contrast medium is visible. Textural features extracted from CESM could discriminate benign and malignant breast lesions using the SVM classifier. A fully automatic system such as a diagnostic support tool for the clinicians using an RF classifier outperformed the human reader.

Radiotherapy: Use of AI in radiotherapy include segmentation of structures used for planning the treatment, such as catheters in brachytherapy. A co-segmentation method to integrate the segmented Biological Target Volume (BTV), using Methionine-PET images, and the segmented Gross Target Volume (GTV), on the respective co-registered MR images, was proposed. Tumor motion increases uncertainty of radiotherapy delivery. A prediction model was implemented to predict tumor motion trajectories from cineMRI (4D-MRI) acquired with a 3 T scanner, which apply SVM to anatomy landmark positions. A custom ML algorithm was developed to outline prostate with pretreatment CT to automatically determine target displacement during RT.

Fig. 2. Examples of activation maps calculated from chest radiographs of patients negative (a) and positive (b) for COVID related pneumonia. ** Image referred from research paper: Artificial intelligence applications in medical imaging: A review of the medical physics research in Italy



AI is playing a significant role in medical imaging research and diagnosis. It is changing the way people process enormous numbers of images. A revolution has taken place in image processing technology using DL. By providing radiologists with the knowledge of basic principles of ML/DL systems, the characteristics of datasets to train them, and their limitations can benefit the future of medical imaging diagnosis. The future of AI in medical imaging is dependent on how we overcome challenges like security and optimization with different clinical applications and once we are successful in overcoming these challenges, using this technology will save time, effort and cost for the overall healthcare community as well as patients. The time to work for and with AI in radiology is now.





  3. Artificial intelligence applications in medical imaging: A review of the medical physics research in Italy