This post is from Oge Marques, PhD and Professor of Engineering and Computer Science at FAU. Oge is a Sigma Xi Distinguished Speaker, book author, and AAAS Leshner Fellow. He also happens to be a MATLAB aficionado and has been using MATLAB in his classroom for more than 20 years. You can follow him on Twitter (@ProfessorOge).

The field of computational pathology (CPATH) consists of using algorithms to analyze digital images obtained through scanning slides of cells and tissues. In recent years, deep learning algorithms that show comparable performance to trained pathologists have been developed for several classification, regression, and segmentation tasks, such as tumor detection and grading.

Applying deep learning (DL) techniques to analyze histopathological tissue sections will start just like any other deep learning application (making sure to adopt a principled workflow and running a large number of experiments in a structured way) but will have a significant additional complicating factor: the need to acquire, label, store, display, and process gigapixel-sized images.

In this blog post we’ll go through an overview of the latest developments in deep learning for CPATH and show you how to handle very large images using MATLAB.

An overview of deep learning in computational pathology

Figure 1: Deep learning workflow in computation pathology.

Figure 1 provides an overview of the deep learning workflow in computation histopathology. Essentially, a deep neural network is trained using patches that must be extracted from a gigapixel-sized whole-slide image (WSI). The choices of architecture (convolutional neural networks (CNNs), fully convolutional networks (FCNs), recurrent neural networks (RNNs), autoencoders, or generative adversarial networks (GANs)) and learning paradigm (supervised, weakly supervised, fully unsupervised, transfer learning) depend on whether the images are labeled and the histopathological image analysis task at hand: CNNs and FCNs are the most widely used architectures for (weakly) supervised learning tasks whereas autoencoders and GANs are popular choices under the unsupervised learning paradigm.

• Supervised learning methods in CPATH have been used for: classification tasks, e.g., predicting whether a patch should be labeled as healthy or cancerous; regression, e.g., detection or localization of cells in histopathology images; and segmentation of structures from histology images.
• Weakly supervised learning techniques exploit coarse-grained (image-level) annotations (e.g., cancer or non-cancer) to automatically infer fine-grained (pixel/patch-level) information, therefore reducing the annotation burden on a pathologist. The most popular paradigm in this category is multiple-instance learning (MIL), in which a training set consists of bags, WSIs labeled as positive or negative; and each bag includes many instances, image patches whose label is to be predicted or unknown. The main goal is to train a classifier to predict both bag-level and instance-level labels, while only bag-level labels are given in the training set.
• Unsupervised (lately rebranded as “self-supervised”) learning is still a young field within deep learning and the applications to CPATH are just starting to appear in the literature.
• Transfer learning approaches are the most widely adopted in histopathology, typically using pretrained (using ImageNet images) models such as VGGNet, Inception, ResNet, MobileNet, and DenseNet. These pretrained models have been extensively used in several cancer grading and prognosis tasks, including public challenges in the field, such as BACH or CAMELYON.

Challenges in acquiring and processing whole-slide images

Regardless of the architecture, learning scheme, or application, digital pathology solutions typically require acquiring and processing a significant number of very large (gigapixel-sized) whole slide images (WSIs), whose contents are often analyzed based on smaller patches (or blocks).

The challenges with acquiring WSIs include:

• Data availability: there are relatively few publicly available data sets in this field (e.g., CAMELYON, HER2, BACH) and much of the published research employs proprietary WSI data sets. The Digital Pathology Association (DPA) maintains a website with a list of image repositories.
• Image format: different scanners output the images using different proprietary file formats (PFFs), creating additional difficulties for data exchange, archiving, and online publication. The lack of a universal image format brings additional costs and potential delays to the curation of large data sets. There have been discussions toward wide adoption of a single open source file format, including the possibility of adopting the DICOM standard for whole slide image encoding.
• Image size: once you successfully acquire enough histopathology images and convert them to a useful format (e.g., TIFF), you must be prepared for the fact that each image file will typically be in the order of a few GB, and plan for the associated implications (such as storage space and network upload/download speeds).

The challenges with processing WSIs include:

• Memory: even a single WSI might be too large to fit entirely into memory.
• Display: ideally you should be able to display the contents of a WSI with zoom/pan/scroll capabilities in monitors whose resolutions are a fraction of the image pixel count.
• Blocks / patches: there should be an elegant way to represent individual blocks (patches) within the image – and treat them as “subimages” whenever needed.
• Artifacts: different artifacts might be present in the WSIs, due to the slide preparation workflow (e.g., color variability in the staining process) or the scanner setup (e.g., different illumination and resolution settings).

The deep learning workflow for CPATH

Figure 2: Simplified deep learning workflow for computational pathology.

Figure 2 shows a simplified deep learning workflow for CPATH. It follows the basic steps of a classic machine learning (ML) / deep learning workflow, with a few notable exceptions and peculiarities:

• The data acquisition process involves collecting tissue specimen, slicing it, extracting each tissue slide, and digitizing it, generating a whole-slide image (WSI). If the resulting image doesn’t pass a quality check, (part of) the acquisition process might have to be repeated.
• The preprocessing step consists of extracting a small number of patches from the gigapixel-sized WSIs. This approach to reducing the high dimensionality of WSIs can be seen as “human-guided feature selection.” Image patches are usually square regions whose size can vary from 32 × 32 pixels up to 10,000 × 10,000 pixels (256 × 256 pixels is a typical patch size). Additionally, this step might include provisions for handling tissue and artifact detection and color management (see example below).
• The modeling block consists of training a selected deep learning model under a selected learning paradigm (e.g., supervised, weakly supervised, self-supervised, transfer learning).
• The postprocessing block might include morphological operations for improving the quality of the predictions at pixel level, fixing small errors, e.g., by filling gaps.
• Finally, the prediction step checks whether the model worked well. If it didn’t (as indicated by the backward arrows) you might want to either: (1) adjust the model’s hyperparameters and perform other similar steps common to most machine learning (ML) and deep learning (DL) tasks; or (2) revisit the preprocessing steps and improve the quality of the input images used for training the model. After all, since ML/DL models learn from data, we must keep this fine balance in mind when refining a solution to better meet a target metric of success.

An example

Here is an example of how to use MATLAB to: (1) handle very large images such as WSIs; and (2) pre- and post-process histology images.

Handling WSIs in MATLAB

The first part of this example shows how to read, display, explore, and organize WSIs (and their patches) in MATLAB. Thanks to the recently introduced blockedImage object, it is now possible to handle very large images without running out of memory. A blockedImage  is an image made from discrete blocks (patches), which can be organized and managed using a blockedImageDatastore object and displayed using bigimageshow.

Useful pre- and post-processing operations on WSIs in MATLAB

Since the goal of using deep learning techniques in CPATH is to produce solutions that are clinically translatable, i.e., capable of working across large patient populations, it is advisable to deal with some of the most likely WSI artifacts upfront, thereby increasing the abilities of the resulting model to generalize over image artifacts found in other test sets.

The second part of this example shows examples of preprocessing operations to handle commonly found artifacts in histopathology images as well as postprocessing morphological operations for improving the quality of the results at pixel level. Essentially, this example should help the medical image analysis community to create an image analysis pipeline for WSIs (and, as bonus, the ability to reproduce the code and examples described in a recent paper on this topic) using MATLAB.

It highlights the usefulness of MATLAB (and Image Processing Toolbox) functions such as:

• Image thresholding and filtering: imbinarize, bwareafilt, and imlincomb
• Morphological image processing operations: imclose, imopen, imdilate, imerode, imfill, and strel
• Feature extraction: bwlabel and regionprops
• Visualization: montage, imoverlay, plot and rectangle

  Figures 3 and 4 show examples of results.

Figure 3: Preprocessing example: (left) initial image; (center) result of thresholding operation to separate tissue pixels from glass pixels; (right) result of applying hull filling to capture the full shape and size of the tissue and remove the slide background from further analysis down the pipeline.

Figure 4: Postprocessing example: (top) overlay of hypothetical predictions of the presence of a region of interest (in green) on a renal transplant biopsy WSI; (center) result of using morphological algorithms to fill holes and remove spurious pixels; (bottom) result of postprocessing (by patch).  

Key takeaways

Deep Learning solutions for computational histopathology require the ability to handle whole slide images, which – in addition to being usually much larger than their images used in other areas of image analysis and computer vision – might suffer from artifacts that could impact the quality of the overall solution. In this blog post we have used MATLAB to show how to handle and process gigapixel-sized WSIs in the context of a CPATH deep learning workflow.

CPATH is an active research area, and new developments and applications of deep learning in this area will likely emerge in the near future. If you’re interested in learning more about CPATH and related issues, I recommend you to check this resource: Classify Large Multiresolution Images Using blockedImageand Deep Learning.