{"id":8854,"date":"2022-01-03T10:19:20","date_gmt":"2022-01-03T15:19:20","guid":{"rendered":"https:\/\/blogs.mathworks.com\/deep-learning\/?p=8854"},"modified":"2022-01-03T10:19:20","modified_gmt":"2022-01-03T15:19:20","slug":"deep-learning-for-computer-vision-using-python-and-matlab","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/deep-learning\/2022\/01\/03\/deep-learning-for-computer-vision-using-python-and-matlab\/","title":{"rendered":"Deep Learning for Computer Vision using Python and MATLAB"},"content":{"rendered":"This post is from\u00a0Oge Marques, PhD<\/a>\u00a0and Professor of Engineering and Computer Science at FAU. Oge is a\u00a0Sigma Xi Distinguished Speaker<\/a>,\u00a0book author<\/a>, and\u00a0AAAS Leshner Fellow<\/a>.\u00a0He also happens to be a MATLAB aficionado and has been using MATLAB in his classroom for more than 20 years. You can follow him on Twitter (@ProfessorOge<\/a>). In this blog post, Oge will cover how to do Deep Learning using both Python and MATLAB for a Computer Vision example.<\/em>\r\n
<\/h6>\r\nDeep Learning (DL) techniques have changed the field of computer vision significantly during the last decade, providing state-of-the-art solutions for classical tasks (e.g.,\u00a0object detection<\/a>\u00a0and\u00a0image classification<\/a>) and opening the doors for solving challenging new problems, such as\u00a0image-to-image translation<\/a>\u00a0and\u00a0visual question answering (VQA)<\/a>.\r\n
<\/h6>\r\nThe success and popularization of DL in computer vision and related areas (e.g., medical image analysis) has been fostered, in great part, by the availability of rich tools, apps and frameworks in the\u00a0Python<\/a>\u00a0and\u00a0MATLAB<\/a>\u00a0ecosystems.\r\n
<\/h6>\r\nIn this blog post, I will show how your team can use both MATLAB and Python effectively and provide an easy-to-follow recipe that you should allow you to leverage \u201cthe best of both worlds\u201d when building computer vision solutions using deep learning.\r\n
<\/h6>\r\n

Background<\/h1>\r\n

<\/h6>\r\nPython<\/a>\u00a0is a programming language created by Guido van Rossum in the early 1990s. It has been adopted by many data scientists and machine\/deep learning researchers thanks to popular packages (e.g.,\u00a0scikit-learn<\/a>) and frameworks (e.g.,\u00a0Keras<\/a>,\u00a0TensorFlow<\/a>,\u00a0PyTorch<\/a>).\r\n
<\/h6>\r\nMATLAB<\/a>\u00a0is a programming and scientific computing platform used to analyze data, develop algorithms, and create models in a variety of fields of science and engineering. It has a successful history of widespread adoption by engineers and researchers in industry and academia. It features many specialized\u00a0toolboxes<\/a>\u00a0which encapsulate relevant algorithms, interactive tools, and rich examples in areas such as\u00a0machine learning<\/a>,\u00a0deep learning<\/a>,\u00a0image processing<\/a>, and\u00a0computer vision<\/a>\u00a0(to mention but a few). MATLAB also has a vibrant\u00a0community of users<\/a>\u00a0who contribute additional functionality (including\u00a0apps<\/a>\u00a0and entire\u00a0toolboxes<\/a>) and a growing presence in popular code-sharing repositories such as\u00a0GitHub<\/a>.\r\n
<\/h6>\r\nIn my personal experience, I have used both MATLAB (for 25 years and counting) and Python (for less than a decade) in different research projects, classes, bootcamps, and publications, mostly in the context of image processing\/analysis, computer vision, and (more recently) data science, machine learning, and deep learning.\r\n
<\/h6>\r\nI have also worked with multidisciplinary teams who adopt a variety of tools and are well-versed in diverse skill sets. I know how important it is to promote and facilitate the adoption of a streamlined and well-documented deep learning workflow. I am also a strong proponent of always using the best available tools to get the job done in the best possible way. Fortunately, you can use the two languages together, which we will show next.\r\n
<\/h6>\r\n

Context and scope<\/h1>\r\n

<\/h6>\r\nThe interoperability of MATLAB and Python has been extensively documented in\u00a0videos<\/a>,\u00a0webinars<\/a>, blog posts, and the\u00a0official MATLAB documentation<\/a>. These resources can be extremely valuable when learning\u00a0how to call Python scripts from MATLAB and vice-versa<\/a>.\r\n\r\nSome of the main reasons<\/a>\u00a0for calling MATLAB from Python can be motivated by the need to:\r\n
<\/h6>\r\n
    \r\n \t
  1. Promote code integration among team members and collaborators using different frameworks and tools.<\/li>\r\n \t
  2. Leverage functionality only available in MATLAB, such as\u00a0apps<\/a>\u00a0and toolboxes (including third-party ones contributed by the MATLAB community).<\/li>\r\n<\/ol>\r\n
    <\/h6>\r\nIn this blog post, I focus on\u00a0integrating MATLAB apps into a Python deep learning workflow for computer vision and image analysis tasks<\/strong>, with emphasis on the data preparation stage of the traditional deep learning workflow (Figure 1). More specifically, I show how your team can leverage the rich interactive capabilities of selected MATLAB apps to prepare, label, annotate, and preprocess your data\u00a0before<\/u><\/em>\u00a0using it as the input to your neural network \u2013 and everything else that follows in the traditional deep learning pipeline.\r\n
    <\/h6>\r\n\"\"\r\n
    <\/h6>\r\nFigure 1: Basic deep learning workflow.\r\n
    <\/h6>\r\nI will assume that: (1) you have a deep learning pipeline for computer vision in Python that you plan to adapt and reuse for a new (set of) task(s); and (2) the images associated with the new task(s) will require interactive actions, such as annotation, labeling, and segmentation.\r\n
    <\/h6>\r\n

    The basic recipe<\/h1>\r\n

    <\/h6>\r\nAssuming that you have\u00a0MATLAB installed and configured in your machine<\/a>\u00a0and your favorite Python setup (e.g., using Jupyter notebooks), calling MATLAB from a Python script is a straightforward process, whose main steps are:\r\n
    <\/h6>\r\n
      \r\n \t
    1. (In MATLAB)<\/em>\u00a0Install the\u00a0MATLAB Engine API for Python<\/a>, which provides a Python package called matlab that allows you to call MATLAB functions and exchange data between Python and MATLAB.<\/li>\r\n \t
    2. (In Python)<\/em>\u00a0Configure paths and working directory.<\/li>\r\n \t
    3. (In Python)<\/em>\u00a0Start a new MATLAB process in the background:<\/li>\r\n<\/ol>\r\nimport matlab.engine<\/code>\r\n\r\neng = matlab.engine.start_matlab('-desktop')<\/code>\r\n
        \r\n \t
      1. (In Python)<\/em>\u00a0Set up your variables (e.g., path to image folders).<\/li>\r\n \t
      2. (In Python)<\/em>\u00a0Call a MATLAB app of your choice (e.g., Image Labeler app).<\/li>\r\n \t
      3. (In MATLAB)<\/em>\u00a0Work (interactively) with the selected app and export results to variables in the workspace.<\/li>\r\n \t
      4. (In Python)<\/em>\u00a0Save the variables needed for the rest of the workflow, e.g., image filenames and associated labels (and their bounding boxes).<\/li>\r\n \t
      5. (In Python)<\/em>\u00a0Use the variables as needed, e.g., processing tabular data using pandas and using image-related labels as ground truth.<\/li>\r\n \t
      6. Repeat steps 3 through 7 as many times as needed in your workflow.<\/li>\r\n \t
      7. (In Python)<\/em>\u00a0Quit the MATLAB engine:<\/li>\r\n<\/ol>\r\neng.exit()<\/code>\r\n

        An example<\/h1>\r\n

        <\/h6>\r\nHere is an\u00a0example<\/a>\u00a0of how to use Python and MATLAB together for two different tasks within the scope of medical image analysis (using deep learning): skin lesion segmentation and (medical) image (ROI) labeling.\r\n
        <\/h6>\r\nDespite the differences among them, each task follows the same basic recipe presented earlier. The specifics of each case are described next.\r\n
        <\/h6>\r\n

        Task A: Skin Lesion Segmentation<\/h2>\r\nThe Task:<\/strong>\u00a0Given a dataset of images containing skin lesions, we want to build a deep learning solution for segmenting each image, i.e., classifying each pixel as belonging to either the lesion (foreground) or the rest of the image (background).\r\n
        <\/h6>\r\nThe Problem:<\/strong>\u00a0In order to train and validate a deep neural network for image segmentation, we need to input both the images as well as the segmentation masks (Figure 2), which are essentially binary images where foreground pixels (in this case corresponding to the lesion) are labeled white and background pixels are marked as black. The job of the network is to learn the segmentation masks for new images.\r\n
        <\/h6>\r\n\"\"\r\n
        <\/h6>\r\nFigure 2: Skin lesion segmentation: input image (left); binary segmentation mask (center);\r\nsegmented image, with green contour outlining the lesion area (right).\r\n
        <\/h6>\r\nThe basic workflow usually consists of using convolutional network architectures, such as\u00a0U-net<\/a>\u00a0and its variations, for which there are multiple examples of implementation in\u00a0Python<\/a>\u00a0and\u00a0MATLAB<\/a>. A crucial component of the solution, however, is the manual creation of the binary masks needed for training and validation. Except for a few publicly available datasets, this time-consuming and specialized task must be performed using a powerful interactive tool.\r\n
        <\/h6>\r\nThe Solution:<\/strong>\u00a0Use the\u00a0MATLAB\u00a0Image Segmenter<\/em>\u00a0app<\/a>\u00a0to create the binary masks and leverage the existing (Python, for the sake of this example) workflow for everything else. Image Segmenter allows you to create masks manually and provides several (semi-)automatic techniques to speed up the process and refine the results (Figures 3 and 4). Both the final segmentation mask image and the segmented version of the original image can be exported to the MATLAB workspace and\/or saved to disk.\r\n
        <\/h6>\r\n\"\"\r\n
        <\/h6>\r\nFigure 3: Image Segmenter app: loading an image containing a skin lesion.\r\n
        <\/h6>\r\n\"\"\r\n
        <\/h6>\r\nFigure 4: Image Segmenter app: result of applying the Graph Cut algorithm after having selected a few foreground control points (in green) and a single background control point (in red).\u00a0The mask appears overlaid on top of the original image.\r\n
        <\/h6>\r\n

        Task B: (Medical) Image (ROI) Labeling<\/h2>\r\nThe Task:<\/strong>\u00a0In a similar context to Task A, we want to build a deep learning solution for detecting regions of interest (ROIs) in each image, i.e., placing a boundary around each relevant region in the image. The most common ROI will be a lesion; other possible ROIs could include stickers, ruler markers, water bubbles, ink marks, and other artifacts.\r\n
        <\/h6>\r\nThe Problem:<\/strong>\u00a0To train and validate a deep neural network for ROI\/object detection, we need to input both the images as well as the labels and coordinates of the relevant ROIs, which can be expressed as rectangles (most common), polygons, or pixel-based masks (similarly to the masks used in segmentation). The job of the network is to learn the location and labels of the relevant ROIs for new images.\r\n
        <\/h6>\r\nOnce again, similarly to what we saw in Task A, a crucial component of the solution is the manual creation of the ROIs (polygons and labels) needed for training and validation. Except for a few publicly available datasets, this time-consuming and specialized task must be performed using a powerful interactive tool.\r\n
        <\/h6>\r\nThe Solution:<\/strong>\u00a0Use the\u00a0MATLAB\u00a0Image Labeler<\/em>\u00a0app<\/a>\u00a0to create and label the ROIs and leverage the existing workflow for everything else. Image Labeler allows you to create ROI labels of different shape, assign them different names and colors, and provides several algorithms to help automate and speed up the process and refine the results (Figure 5). The resulting ROIs can be exported to the MATLAB workspace and subsequently used as variables in your Python code (see\u00a0example on GitHub<\/a>\u00a0for details).\r\n
        <\/h6>\r\n\"\"\r\n
        <\/h6>\r\nFigure 5: Image Labeler app in the context of dermoscopic images containing artifacts.\r\nThe selected image contains two rectangular ROIs, labeled as\u00a0lesion<\/em>\u00a0and\u00a0sticker<\/em>.\r\n
        <\/h6>\r\n

        Key takeaways<\/h1>\r\n

        <\/h6>\r\nDeep Learning projects are often collaborative endeavors that require using the best tools for the job, enabling effective code integration, development, and testing strategies, promoting communication, and ensuring reproducibility of code. Your team can (and should) leverage the best of MATLAB and Python while developing your deep learning projects. In this blog post I have shown how to use Python and MATLAB together for a few tasks related to computer vision and medical image analysis problems.\r\n
        <\/h6>\r\nIntegration of Python and MATLAB goes significantly beyond the scope of this blog post; check out the resources listed below for more.\r\n
        <\/h6>\r\n

        Learn more about it<\/h1>\r\n

        <\/h6>\r\nThis blog post was inspired by\u00a0recent blog posts by Lucas Garc\u00eda<\/a>\u00a0and a series of great videos by\u00a0Heather Gorr<\/a>, Yann Debray, and colleagues. I strongly encourage you to follow them and check out their very informative examples and tutorials.\r\n\r\nIf you\u2019re interested in other aspects of the deep learning workflow, these are some blog posts in which I:\r\n
        <\/h6>\r\n