{"id":10118,"date":"2022-10-04T15:54:39","date_gmt":"2022-10-04T19:54:39","guid":{"rendered":"https:\/\/blogs.mathworks.com\/deep-learning\/?p=10118"},"modified":"2024-04-16T16:49:33","modified_gmt":"2024-04-16T20:49:33","slug":"whats-new-in-interoperability-with-tensorflow-and-pytorch","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/deep-learning\/2022\/10\/04\/whats-new-in-interoperability-with-tensorflow-and-pytorch\/","title":{"rendered":"What&#8217;s New in Interoperability with TensorFlow and PyTorch"},"content":{"rendered":"For deep learning, MATLAB allows users to create and train models in MATLAB or leverage models trained in open source via model conversion. Prior to MATLAB R2022b, support for model conversion included: import from and export to ONNX\u2122, and import from TensorFlow\u2122. We are excited to share that as of MATLAB R2022b, users can now<strong> export models to TensorFlow<\/strong> as Python\u00ae code and can <strong>import models from PyTorch\u00ae<\/strong> (starting with support for image classification).\r\n<h6><\/h6>\r\n&nbsp;\r\n<p style=\"font-size: 18px;\"><strong>Export to TensorFlow<\/strong><\/p>\r\n<p style=\"text-align: center;\"><img decoding=\"async\" loading=\"lazy\" class=\"wp-image-10187 size-full aligncenter\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2022\/09\/importexport_exportflow.png\" alt=\"\" width=\"420\" height=\"143\" \/><\/p>\r\n\r\n<h6><\/h6>\r\nThe support package <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/64649-deep-learning-toolbox-converter-for-tensorflow-models\">Deep Learning Toolbox Converter for TensorFlow Models<\/a> just added the capability to export from MATLAB to TensorFlow, by using the\u00a0<a href=\"https:\/\/www.mathworks.com\/help\/deeplearning\/ref\/exportnetworktotensorflow.html\"><span style=\"font-family: courier;\">exportNetworkToTensorFlow<\/span><\/a> function.\r\n<h6><\/h6>\r\nThere are many reasons to be excited about the new <span style=\"font-family: courier;\">exportNetworkToTensorFlow<\/span> function:\r\n<h6><\/h6>\r\n<table style=\"height: 173px;\" width=\"678\">\r\n<tbody>\r\n<tr>\r\n<td style=\"padding: 10px; border: 1px solid black; background-color: #f5fff2;\" width=\"208\">\r\n<ol>\r\n \t<li>You can export deep learning networks and layer graphs (e.g., convolutional, LSTM) directly to TensorFlow.<\/li>\r\n \t<li>It\u2019s easy to export a network, load it as a TensorFlow model, and use the exported model for prediction. For a relevant example, see <a href=\"https:\/\/www.mathworks.com\/help\/deeplearning\/ref\/exportnetworktotensorflow.html#mw_b88d248d-a405-4cfa-a1c3-f610dfc41675\">Export Network to TensorFlow and Classify Image<\/a>.<\/li>\r\n \t<li>It\u2019s easy to export an untrained layer graph, load it as a TensorFlow model, and train the exported model. For a relevant example, see <a href=\"https:\/\/www.mathworks.com\/help\/deeplearning\/ref\/exportnetworktotensorflow.html#mw_969bbaaf-2bad-47fd-bba4-e62ff2bb79d3\">Export Untrained Layer Graph to TensorFlow<\/a>.<\/li>\r\n \t<li>You can save the exported model in any standard TensorFlow format, such as SavedModel or HDF5 format, and share it with your colleagues who work in TensorFlow.<\/li>\r\n \t<li>Many MATLAB layers can be converted to TensorFlow layers. For a full list of the layers, see <a href=\"https:\/\/www.mathworks.com\/help\/deeplearning\/ref\/exportnetworktotensorflow.html#mw_5df2ceb2-14b1-4133-9a27-3542d9a517de\">Layers Supported for Export to TensorFlow<\/a>.<\/li>\r\n<\/ol>\r\n<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<h6><\/h6>\r\nHere I am showing the basic workflow of how to export a deep learning network to TensorFlow, load it as a TensorFlow model, and save it in SavedModel format.\r\n<h6><\/h6>\r\n<table style=\"height: 173px;\" width=\"678\">\r\n<tbody>\r\n<tr>\r\n<td style=\"padding: 10px; border: 1px solid black;\" width=\"208\"><span style=\"text-decoration: underline;\">MATLAB Code:<\/span>\r\n<h6><\/h6>\r\nLoad a pretrained network. The\u00a0<a href=\"https:\/\/www.mathworks.com\/help\/deeplearning\/ug\/pretrained-convolutional-neural-networks.html\">Pretrained Deep Neural Networks<\/a> documentation page shows you all options of how to get a pretrained network. You can alternatively create your own network.\r\n<h6><\/h6>\r\n<pre>net = darknet19;<\/pre>\r\n<h6><\/h6>\r\nExport the network <span style=\"font-family: courier;\">net<\/span> to TensorFlow. The <span style=\"font-family: courier;\">exportNetworkToTensorFlow<\/span> function saves the TensorFlow model in the Python package DarkNet19.\r\n<h6><\/h6>\r\n<pre> exportNetworkToTensorFlow(net,\"DarkNet19\")<\/pre>\r\n<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<h6><\/h6>\r\nThe\u00a0DarkNet19\u00a0package contains four files:\r\n<ul>\r\n \t<li>The\u00a0_init_.py\u00a0file, which defines the DarkNet19 folder as a regular Python package.<\/li>\r\n \t<li>The\u00a0model.py\u00a0file, which contains the code that defines the untrained TensorFlow-Keras model.<\/li>\r\n \t<li>The\u00a0README.txt\u00a0file, which provides instructions on how to load the TensorFlow model and save it in HDF5 or SavedModel format.<\/li>\r\n \t<li>The\u00a0weights.h5\u00a0file which contains the model weights in HDF5 format.<\/li>\r\n<\/ul>\r\n<h6><\/h6>\r\n<p style=\"text-align: center;\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-10445 \" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2022\/09\/package_diagram3.png\" alt=\"\" width=\"733\" height=\"457\" \/><\/p>\r\n<p style=\"text-align: center;\"><strong>Figure: The exported TensorFlow model is saved in the regular Python package DarkNet19.<\/strong><\/p>\r\n\r\n<h6><\/h6>\r\n&nbsp;\r\n<table style=\"height: 173px;\" width=\"678\">\r\n<tbody>\r\n<tr>\r\n<td style=\"padding: 10px; border: 1px solid black;\" width=\"208\"><span style=\"text-decoration: underline;\">Python Code:<\/span>\r\n<h6><\/h6>\r\nLoad the exported TensorFlow model from the DarkNet19 package.\r\n<h6><\/h6>\r\n<pre class=\"brush: python\" style=\"background-color: white;\">import DarkNet19\r\nmodel = DarkNet19.load_model()<\/pre>\r\n<h6><\/h6>\r\nSave the exported model in the SavedModel format.\r\n<h6><\/h6>\r\n<pre class=\"brush: python\" style=\"background-color: white;\">model.save(\"DarkNet19_savedmodel\")<\/pre>\r\n<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<h6><\/h6>\r\n&nbsp;\r\n<p style=\"font-size: 18px;\"><strong>Import from PyTorch<\/strong><\/p>\r\n\r\n<h6><\/h6>\r\n<p style=\"text-align: center;\"><img decoding=\"async\" loading=\"lazy\" class=\"wp-image-10298 size-full aligncenter\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2022\/09\/importexport_importflow.png\" alt=\"\" width=\"423\" height=\"228\" \/><\/p>\r\n\r\n<h6><\/h6>\r\nIn R2022b we introduced the <a href=\"https:\/\/www.mathworks.com\/matlabcentral\/fileexchange\/111925-deep-learning-toolbox-converter-for-pytorch-models\">Deep Learning Toolbox Converter for PyTorch Models<\/a> support package. This initial release supports importing image classification models. Support for other model types will be added in future releases.\r\n<h6><\/h6>\r\nUse the <a href=\"https:\/\/www.mathworks.com\/help\/deeplearning\/ref\/importnetworkfrompytorch.html\"><span style=\"font-family: courier;\">importNetworkFromPyTorch<\/span><\/a> function to import a PyTorch model. Make sure that the PyTorch model that you are importing is pretrained and traced. I am showing you here how to import an image classification model from PyTorch and initialize it.\r\n<h6><\/h6>\r\n<table style=\"height: 173px;\" width=\"678\">\r\n<tbody>\r\n<tr>\r\n<td style=\"padding: 10px; border: 1px solid black;\" width=\"208\"><span style=\"text-decoration: underline;\">Python Code:<\/span>\r\n<h6><\/h6>\r\nLoad a pretrained image classification model from the <a href=\"https:\/\/pytorch.org\/vision\/0.8\/models.html\">TorchVision library<\/a>.\r\n<h6><\/h6>\r\n<pre class=\"brush: python\" style=\"background-color: white;\">import torch\r\nfrom torchvision import models\r\nmodel = models.mnasnet1_0(pretrained=True)<\/pre>\r\n<h6><\/h6>\r\nTrace the PyTorch model. For more information on how to trace a PyTorch model,\r\n\r\ngo to <a href=\"https:\/\/pytorch.org\/docs\/stable\/generated\/torch.jit.trace.html\">Torch documentation: Tracing a function<\/a>.\r\n<h6><\/h6>\r\n<pre class=\"brush: python\" style=\"background-color: white;\">X = torch.rand(1,3,224,224)\r\ntraced_model = torch.jit.trace(model.forward,X)\r\n<\/pre>\r\n<h6><\/h6>\r\nSave the PyTorch model.\r\n<h6><\/h6>\r\n<pre class=\"brush: python\" style=\"background-color: white;\">traced_model.save(\"traced_mnasnet1_0.pt\")\r\n<\/pre>\r\n<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<h6><\/h6>\r\n<table style=\"height: 173px;\" width=\"678\">\r\n<tbody>\r\n<tr>\r\n<td style=\"padding: 10px; border: 1px solid black;\" width=\"208\"><span style=\"text-decoration: underline;\">MATLAB Code:<\/span>\r\n<h6><\/h6>\r\nImport the PyTorch model into MATLAB by using the <span style=\"font-family: courier;\">importNetworkTFromPyTorch<\/span> function. The function imports the model as an uninitialized <span style=\"font-family: courier;\">dlnetwork<\/span> object without an input layer.\r\n<h6><\/h6>\r\n<pre>net = importNetworkFromPyTorch(\"traced_mnasnet1_0.pt\");\r\n<\/pre>\r\nSpecify the input size of the imported network and create an image input layer. Then, add the image input layer to the imported network and initialize the network by using the\r\n<a href=\"https:\/\/www.mathworks.com\/help\/deeplearning\/ref\/dlnetwork.addinputlayer.html\"><span style=\"font-family: courier;\">addInputLayer<\/span><\/a> function (also new in R2022b).\r\n<h6><\/h6>\r\n<pre>InputSize = [224 224 3];\r\nInputLayer = imageInputLayer(InputSize,Normalization=\"none\");\r\nnet = addInputLayer(net,InputLayer,Initialize=true);\r\n<\/pre>\r\n<\/td>\r\n<\/tr>\r\n<\/tbody>\r\n<\/table>\r\n<h6><\/h6>\r\nYou might have noticed in the above code that the input dimensions in PyTorch and MATLAB have a different order. For more information, see <a href=\"https:\/\/www.mathworks.com\/help\/deeplearning\/ug\/tips-on-importing-models-from-tensorflow-pytorch-and-onnx.html#mw_ca5d4cba-9c12-4f01-8fe1-6329730c92b2\">Input Dimension Ordering for Deep Learning Platforms<\/a>.\r\n<h6><\/h6>\r\nFor more details on how to import a PyTorch model, how to initialize the imported model, and how to perform workflows (such as prediction and training) on the imported model, see the <a href=\"https:\/\/www.mathworks.com\/help\/deeplearning\/ref\/importnetworkfrompytorch.html#mw_d51e64bc-e9ad-409d-ba4a-d70c47f40449\">Examples<\/a> of the <span style=\"font-family: courier;\">importNetworkTFromPyTorch<\/span> documentation page.\r\n<h6><\/h6>\r\n&nbsp;\r\n<p style=\"font-size: 18px;\"><strong>Interoperability Capabilities Summary<\/strong><\/p>\r\n<p style=\"text-align: center;\"><img decoding=\"async\" loading=\"lazy\" class=\"alignnone wp-image-10343 size-full\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2022\/09\/import_export_platforms.png\" alt=\"\" width=\"463\" height=\"286\" \/><\/p>\r\n\r\n<h6><\/h6>\r\nThe interoperability support packages allow you to connect Deep Learning Toolbox with TensorFlow, Pytorch, and ONNX. Use the import and export functions to access models available in open-source repositories and collaborate with colleagues who work in other deep learning frameworks.\r\n<h6><\/h6>\r\n<u>More information:<\/u>\r\n<ul>\r\n \t<li>To find all the available import and export functions (and their documentation), go to <a href=\"https:\/\/www.mathworks.com\/help\/deeplearning\/deep-learning-import-and-export.html\">Deep Learning Import and Export<\/a>.<\/li>\r\n \t<li>To learn more about how to import and export networks, see <a href=\"https:\/\/www.mathworks.com\/help\/deeplearning\/ug\/interoperability-between-deep-learning-toolbox-tensorflow-pytorch-and-onnx.html\">Interoperability Between Deep Learning Toolbox, TensorFlow, PyTorch, and ONNX<\/a>.<\/li>\r\n \t<li>For answers to common questions about importing models, see <a href=\"https:\/\/www.mathworks.com\/help\/deeplearning\/ug\/tips-on-importing-models-from-tensorflow-pytorch-and-onnx.html\">Tips on Importing Models from TensorFlow, PyTorch, and ONNX<\/a>.<\/li>\r\n \t<li>If you are working just in MATLAB, you can probably find a suitable network in our constantly-updated\u00ad\u00ad\u00ad\u00ad model repository: <a href=\"https:\/\/github.com\/matlab-deep-learning\/MATLAB-Deep-Learning-Model-Hub\">MATLAB Deep Learning Model Hub<\/a>.<\/li>\r\n<\/ul>\r\n<h6><\/h6>","protected":false},"excerpt":{"rendered":"<div class=\"overview-image\"><img decoding=\"async\"  class=\"img-responsive\" src=\"https:\/\/blogs.mathworks.com\/deep-learning\/files\/2022\/09\/importexport_exportflow.png\" onError=\"this.style.display ='none';\" \/><\/div><p>For deep learning, MATLAB allows users to create and train models in MATLAB or leverage models trained in open source via model conversion. Prior to MATLAB R2022b, support for model conversion... <a class=\"read-more\" href=\"https:\/\/blogs.mathworks.com\/deep-learning\/2022\/10\/04\/whats-new-in-interoperability-with-tensorflow-and-pytorch\/\">read more >><\/a><\/p>","protected":false},"author":194,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[9,39,45,42],"tags":[],"_links":{"self":[{"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/posts\/10118"}],"collection":[{"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/users\/194"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/comments?post=10118"}],"version-history":[{"count":108,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/posts\/10118\/revisions"}],"predecessor-version":[{"id":14673,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/posts\/10118\/revisions\/14673"}],"wp:attachment":[{"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/media?parent=10118"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/categories?post=10118"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.mathworks.com\/deep-learning\/wp-json\/wp\/v2\/tags?post=10118"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}