In this blog post, my colleague Dave Schowalter will introduce you to a new ecosystem that combines the photo-realistic simulation capabilities from NVIDIA Isaac Sim^TM and the sensor processing and AI modeling capabilities from MathWorks for building realistic robot simulations.

Learn more at the joint NVIDIA/MathWorks webinar on September 12, “MATLAB and Isaac Sim”.

I am Dave Schowalter! As a Partner Programs Manager at MathWorks, I build technology partnerships in the robotics and autonomous systems industry.

Simulating autonomous robots in a photo-realistic environment provides several practical advantages, especially when incorporating sensors and perception in the models. It enables robotics engineers and researchers to thoroughly assess robot performance in diverse, complex settings, enhancing adaptability and problem-solving capabilities.

Autonomous robots use machine learning and sensor-based perception algorithms. Incorporating sensors and sensor data processing with a photo-realistic scene simulation helps with improving accuracy and performance of these algorithms. This also becomes an advantage when training an autonomous robot based upon synthetic sensor outputs.

NVIDIA Isaac Sim and MathWorks Model-Based Design together provide an integrated approach to create and perform these simulations. It offers an efficient platform for addressing safety concerns and refining sensor calibration, ultimately leading to more reliable real-world implementations.

With Isaac Sim^TM, NVIDIA has created a high level of photo-realism by using a combination of techniques, including:

• Real-time Ray Tracing: Ray tracing is a technique that simulates the way light interacts with objects in the real world. This allows Isaac Sim to create realistic reflections, refractions, and shadows.
• Physically-based Rendering: Physically-based rendering is a technique that uses the laws of physics to simulate the way light interacts with materials. This allows Isaac Sim to replicate the way actual materials such as metal, plastic, and wood reflect and scatter light.
• High-Quality Textures: Isaac Sim uses high-quality textures to give objects a genuine appearance. These textures are created using a variety of methods, including scanning real objects and generating them using computer graphics techniques.
• Advanced Lighting: Isaac Sim uses advanced lighting techniques to create authentic lighting conditions. These techniques include global illumination, which simulates the way light bounces off objects, and ambient occlusion, which simulates the way shadows are created by objects blocking light.
• High-Performance Rendering: Isaac Sim uses NVIDIA’s GPUs to render scenes in real time. This allows users to perceive the interaction of simulation assets with the environment in a natural way.

Although Isaac Sim offers advanced physics simulation to recreate the behavior of objects in the real world, manipulation of synthetic data from multiple sensors (“sensor fusion”) and the use of those results to train AI algorithms and determine robot behavior can all be designed and managed in MATLAB and Simulink.  Such a system built in Simulink is depicted below.

Once designed, the entire robot behavior in its environment can be predicted and then adjusted as needed, before testing a physical prototype.  This integration of MATLAB and Simulink with Isaac Sim is most efficiently implemented through ROS, using the ROS Toolbox add-on in MATLAB.  A screenshot of the integration in use is shown below.

Finally, the program can be deployed on embedded hardware (for example on the NVIDIA Jetson platform) to drive the end application.

If you are interested in learning more about this workflow, please register for the joint NVIDIA/MathWorks webinar on September 12, “MATLAB and Isaac Sim.”  In the webinar, you will learn how to use this integration with workflows for manipulator and mobile robot applications.