{"id":8259,"date":"2021-12-02T09:06:13","date_gmt":"2021-12-02T14:06:13","guid":{"rendered":"https:\/\/blogs.mathworks.com\/deep-learning\/?p=8259"},"modified":"2021-12-02T09:06:13","modified_gmt":"2021-12-02T14:06:13","slug":"synthetic-image-generation-using-gans","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/deep-learning\/2021\/12\/02\/synthetic-image-generation-using-gans\/","title":{"rendered":"Synthetic Image Generation using GANs"},"content":{"rendered":"This post is from\u00a0Oge Marques, PhD<\/a>\u00a0and Professor of Engineering and Computer Science at FAU. Oge is a\u00a0Sigma Xi Distinguished Speaker<\/a>,\u00a0book author<\/a>, and\u00a0AAAS Leshner Fellow<\/a>.\u00a0He also happens to be a MATLAB aficionado and has been using MATLAB in his classroom for more than 20 years. You can follow him on Twitter (@ProfessorOge<\/a>).<\/em>\r\n
<\/h6>\r\nOccasionally a novel neural network architecture comes along that enables a truly unique way of solving specific deep learning problems. This has certainly been the case with Generative Adversarial Networks (GANs), originally proposed by Ian Goodfellow et al. in\u00a0a 2014 paper<\/a>\u00a0that has been cited more than 32,000 times since its publication. Among other applications, GANs have become the preferred method for synthetic image generation. The results of using GANs for creating realistic images of\u00a0people who do not exist<\/a>\u00a0have raised many\u00a0ethical issues<\/a>\u00a0along the way.\r\n
<\/h6>\r\nIn this blog post we focus on using GANs to generate synthetic images of skin lesions for medical image analysis in dermatology.\r\n\r\n\"\"\r\n\r\nFigure 1: How a generative adversarial network (GAN) works.<\/em>\r\n
<\/h6>\r\n

A Quick GAN Lesson<\/h2>\r\nEssentially, GANs consist of two neural network agents\/models (called\u00a0generator<\/em>\u00a0and\u00a0discriminator<\/em>) that compete with one another in a zero-sum game, where one agent's gain is another agent's loss. The\u00a0generator<\/em>\u00a0is used to generate new plausible examples from the problem domain whereas the\u00a0discriminator<\/em>\u00a0is used to classify examples as real (from the domain<\/em>) or fake (generated<\/em>). The discriminator is then updated to get better at discriminating real and fake samples in subsequent iterations, and the generator is updated based on how well the generated samples fooled the discriminator (Figure 1).\r\n
<\/h6>\r\nDuring its history, numerous architectural variations and improvements over the original GAN idea have been proposed in the literature. Most GANs today are at least loosely based on the DCGAN (Deep Convolutional Generative Adversarial Networks) architecture, formalized by\u00a0Alec Radford, Luke Metz and Soumith Chintala in\u00a0their 2015 paper<\/a>.\r\n
<\/h6>\r\nYou\u2019re likely to see DCGAN,\u00a0LAPGAN<\/a>, and\u00a0PGAN<\/a>\u00a0used for unsupervised techniques like image synthesis, and\u00a0cycleGAN<\/a>\u00a0and\u00a0Pix2Pix<\/a>\u00a0used for cross-modality image-to-image translation.\r\n
<\/h6>\r\n

GANs for Medical Images<\/h2>\r\nThe use of GANs to create synthetic medical images is motivated by the following aspects:\r\n

<\/h6>\r\n
    \r\n \t
  1. Medical (imaging) datasets are heavily unbalanced, i.e., they contain many more images of healthy patients than any pathology. The ability to create synthetic images (in different modalities) of specific pathologies could help alleviate the problem and provide more and better samples for a deep learning model to learn from.<\/li>\r\n \t
  2. Manual annotation of medical images is a costly process (compared to similar tasks for generic everyday images, which could be handled using crowdsourcing or\u00a0smart image labeling tools<\/a>). If a GAN-based solution were reliable enough to produce appropriate images requiring minimal labeling\/annotation\/validation by a medical expert, the time and cost savings would be appealing.<\/li>\r\n \t
  3. Because the images are synthetically generated, there are no patient data or privacy concerns.<\/li>\r\n<\/ol>\r\nSome of the main\u00a0challenges<\/strong>\u00a0for using GANs to create synthetic medical images, however, are:\r\n
    <\/h6>\r\n
      \r\n \t
    1. Domain experts would still be needed to assess quality of synthetic images while the model is being refined, adding significant time to the process before a reliable synthetic medical image generator can be deployed.<\/li>\r\n \t
    2. Since we are ultimately dealing with patient health, the stakes involved in training (or fine-tuning) predictive models using synthetic images are higher than using similar techniques for non-critical AI applications. Essentially, if models learn from data, we must trust the data that these models are trained on.<\/li>\r\n<\/ol>\r\nThe popularity of using GANs for medical applications has been growing at a fast pace in the past few years. In addition to synthetic image generation in a variety of medical domains, specialties, and image modalities, other applications of GANs such as cross-modality image-to-image translation (usually among MRI, PET, CT, and MRA) are also being researched in prominent labs, universities, and research centers worldwide.\r\n
      <\/h6>\r\nIn the field of dermatology, unsupervised synthetic image generation methods have been used to create high resolution synthetic skin lesion samples, which have also been successfully used in the training of skin lesion classi\ufb01ers.\u00a0State-of-the-art (SOTA) algorithms<\/a>\u00a0have been able to synthesize high resolution images of skin lesions which expert dermatologists could not reliably tell apart from real samples. Figure 2 shows examples of synthetic images generated by a\u00a0recently published solution<\/a>\u00a0as well as real images from the training dataset.\r\n\r\n\"\"\r\n\r\nFigure 2: (L) synthetically generated images using\u00a0state-of-the-art techniques<\/a>;\r\n(R) actual skin lesion images from a typical training dataset.<\/em>\r\n
      <\/h6>\r\n

      An example<\/h2>\r\nHere is an\u00a0example<\/a>\u00a0of how to use MATLAB to generate synthetic images of skin lesions.\r\n
      <\/h6>\r\nThe training dataset consists of annotated images from the\u00a0ISIC 2016 challenge<\/a>, Task 3 (Lesion classification<\/em>) data set, containing 900 dermoscopic lesion images in JPEG format.\r\n
      <\/h6>\r\nThe code is based on\u00a0an example<\/a>\u00a0using a more generic dataset, and then customized for medical images. It highlights MATLAB\u2019s\u00a0recently added capabilities<\/a>\u00a0for handling more complex deep learning tasks, including the ability to:\r\n
      <\/h6>\r\n