bio_img_autonomous-systems

Autonomous Systems

Design, develop, and test autonomous systems with MATLAB

From Embodied Reasoning to Reliable Robot Actions with MATLAB

Posted by YJ Lim,

10 views (last 30 days)  |   Likes   |   0 comment

1  Combining External AI Reasoning with Trusted Robotics Engineering Tools

Last year, I wrote about the potential of vision-language-action, or VLA, models for robotics. These models can connect visual observations and natural-language instructions directly to robot actions, combining perception, planning, and action generation in an end-to-end learned system (VLA Blog).

That approach remains promising, particularly for adaptable and dexterous robot behavior. But does one learned model always need to do everything?

 

A complementary architecture is emerging: use external VLM-based embodied reasoning models for visual understanding, spatial reasoning, task planning, and task verification, while established robotics tools handle motion planning, control, simulation, and deployment.

Embodied AI reasons about what the robot should do. MATLAB and Simulink determine how the robot can do it reliably.

Simulation showing a robot sorting pipe fittings by color in one task and by shape in another, placing parts on the left or right table according to natural-language instructions.

2 From End-to-End Action to Embodied Reasoning

Many embodied reasoning models build on vision-language model, or VLM, capabilities. A VLM connects visual observations with natural-language understanding. For robotics, embodied reasoning extends this capability to spatial reasoning, task decomposition, and task verification. Unlike a VLA model, however, a VLM-based reasoning model does not necessarily generate low-level robot actions.

 

Consider the instruction:  “Move the elbow fittings to the table on the left.”

An embodied reasoning model can analyze the instruction and camera observations to determine:

  • Which objects are elbow fittings
  • Which surface is the left table
  • What order the objects should be moved
  • Where each object should be placed
  • Whether the task was completed successfully

Rather than generating every joint command directly, the model can produce task-level outputs such as object selections, pick-and-place targets, and completion criteria.

Robotics engineering tools can then translate those intentions into executable robot motion by performing:

  • Coordinate transformations and inverse kinematics
  • Collision checking and motion planning
  • Trajectory generation and control
  • Simulation and system-level testing
  • Robot connectivity and deployment

This creates a useful separation of responsibilities.

  • The embodied reasoning model provides semantic and task-level intelligence.
  • The robotics engineering tools provides geometric, kinematic, and execution-level intelligence.

 

3 A Closed-Loop Robotics Workflow

The architecture can be viewed as two connected layers.

The embodied reasoning layer

An external model such as Gemini Robotics-ER or Nvidia Cosmos 3 interprets natural-language instructions and visual observations. It can identify task-relevant objects, understand spatial relationships, decompose the task, monitor progress, and determine whether the requested objective has been achieved.

This layer answers questions such as:

  • What does the user want?
  • Which objects are relevant?
  • What is the appropriate task sequence?
  • Has the task been completed correctly?
  • Does the plan need to change?

The robotics execution layer

MATLAB, Simulink, and Robotics System Toolbox translate task-level intentions into feasible robot behavior.

This layer can perform:

  • Robot and environment modeling
  • Coordinate-frame transformations
  • Forward and inverse kinematics
  • Collision checking
  • Manipulator motion planning
  • Trajectory generation
  • Joint and task-space control
  • Simulation and visualization
  • Connectivity to robot hardware
  • Code generation and deployment

 

This layer answers a different set of questions:

  • Can the robot reach the requested object?
  • Is the proposed grasp configuration valid?
  • Does the motion avoid obstacles and self-collisions?
  • Does it satisfy joint and workspace constraints?
  • What trajectory should the robot follow?
  • How should that trajectory be executed on the hardware?

 

The two layers operate together in a closed loop.

Embodied reasoning interprets instructions and camera input, then uses MATLAB robotics tools for motion planning, control, and closed-loop task verification.

In the intelligent bin-picking example from the accompanying material, Gemini Robotics performs high-level reasoning and task verification. Robotics System Toolbox handles motion planning, inverse kinematics, and control. Simulink integrates the complete closed-loop workflow and supports both simulation and robot hardware.

Today 10:56 AM ER Blog Post Materials.pdf PDF I'd like to write a blog post on GenAI + MATLAB. My focus is on Embodded Reasoning from external (such as Gemini Robotics or Nvidia Cosmos 3) paired with MATLAB Robotics System Toolbox's deterministic robotics tools to provide. Please refer this slide deck included. I'd like to highlight completementary story. Can you draft 1st version Draft 1 From Embodied Reasoning to Reliable Robot Action with MATLAB Generative AI is moving beyond text, images, and code. A new generation of models is beginning to understand physical environments, interpret natural-language instructions, reason about objects and spatial relationships, and plan actions for robots. Google DeepMind’s Gemini Robotics-ER 1.6, for example, is designed for visual and spatial understanding, task planning, progress monitoring, and success detection. NVIDIA Cosmos 3 combines physical-world reasoning with world generation and action-generation capabilities for Physical AI development. These models represent an important step toward robots that can understand instructions such as: “Move the elbow fittings to the table on the left.” A conventional robot program would need predefined object classes, task logic, target locations, and execution sequences. An embodied reasoning model can instead interpret the instruction together with camera observations, identify the relevant objects and destination, determine an appropriate task sequence, and monitor whether the task was completed. But understanding what the robot should do is only part of the problem. The robot must still determine how to do it while respecting its geometry, joint limits, obstacles, actuator constraints, controller behavior, and execution environment. This is where external embodied reasoning models and established robotics engineering tools become complementary. One Model Does Not Need to Do Everything There is a tendency to view foundation models as replacements for conventional robotics algorithms. A more practical architecture assigns each technology the responsibilities it handles best. An embodied reasoning model can provide the semantic and task-level intelligence needed to: Interpret natural-language instructions Understand objects and spatial relationships Select objects and destinations Determine task order Monitor execution progress Detect task completion or failure Replan when the environment changes MATLAB and Robotics System Toolbox can then provide the algorithmic and constraint-aware execution capabilities needed to: Represent the robot and its environment Solve forward and inverse kinematics Check collisions Plan feasible manipulator motions Generate smooth trajectories Apply control algorithms Simulate the complete behavior Connect to and deploy on robot hardware Robotics System Toolbox includes algorithms for collision checking, path planning, trajectory generation, forward and inverse kinematics, dynamics, mapping, localization, path following, and motion control. The resulting division of responsibility is straightforward: Embodied reasoning determines what should happen. MATLAB helps determine how it can happen reliably. From Semantic Intent to Executable Motion Consider an intelligent bin-picking application. The system receives a camera image and the instruction: “Move the elbow fittings to the left table.” An embodied reasoning model first interprets the scene. It identifies the elbow fittings, distinguishes them from other parts, understands which surface is the left table, and creates a task-level plan. The output should not be sent directly to the robot actuators. Instead, it becomes an input to the robot execution workflow. Robotics System Toolbox can use the selected object and target information to calculate candidate grasp and placement configurations. Inverse kinematics determines whether the robot can reach those configurations. Collision checking and motion-planning algorithms determine whether the robot can move between them without intersecting the bin, table, surrounding equipment, or itself. Trajectory-generation algorithms then convert the planned path into smooth, time-parameterized robot motion. Simulink can coordinate these components in a closed-loop system that includes the reasoning model, motion planning, control, simulation environment, sensors, and hardware interfaces. The intelligent bin-picking architecture in the accompanying slide deck follows this pattern. Gemini Robotics performs high-level task reasoning and task verification, while Robotics System Toolbox performs motion planning and control. Camera feedback closes the loop so the reasoning model can monitor progress, verify object placement, and request replanning when necessary. The same architecture can run with a simulated robot or connect to physical hardware. Why Embodied Reasoning Changes the Perception Workflow Traditional deep-learning-based robot perception can require several separately engineered stages: Collect or generate training images. Label objects and poses. Train an object detector or segmentation network. Estimate six-degree-of-freedom object poses. Convert perception outputs into task logic. Pass target poses to motion-planning and control algorithms. This architecture remains valuable, particularly when high accuracy and predictable behavior are required for a well-defined application. However, each new object, task, or environment may require additional data, training, tuning, and integration. Embodied reasoning models can simplify the task-level portion of this pipeline. Rather than training an individual model for every variation of a sorting task, the system can use visual observations and a natural-language instruction such as: “Sort these parts by color.” “Place all cross fittings on the right table.” “Move the damaged parts into the inspection area.” The model reasons about the requested objective and the observed environment. MATLAB then converts that semantic objective into geometric and executable robot actions. Embodied reasoning does not eliminate the need for calibration, coordinate transformations, kinematics, motion planning, control, or validation. Instead, it provides a more flexible front end for specifying and adapting robot behavior. Closing the Loop with Verification The most useful embodied reasoning architecture is not a one-time sequence in which the model creates a plan and the robot blindly follows it. Physical environments change. An object may move during grasping. The gripper may fail to pick up a part. A destination may be occupied. The robot may reach a configuration from which the next action is infeasible. A closed-loop workflow continuously returns observations and execution status to the reasoning model. The model can then evaluate questions such as: Was the intended object successfully grasped? Was it placed at the requested destination? Are there additional objects to process? Has the scene changed? Should the task plan be revised? At the same time, the robotics execution layer continues to evaluate geometric and dynamic feasibility. A semantically correct instruction can still produce an unreachable target or a path that violates a constraint. The execution layer can reject that request, select an alternative configuration, or return structured feedback to the reasoning model. This creates two complementary verification loops: Task-level verification evaluates whether the robot is accomplishing the user’s intent. Engineering-level verification evaluates whether the requested motion is feasible and satisfies the system’s constraints. Together, these loops provide a more robust architecture than either embodied reasoning or conventional robot programming alone. What About Vision-Language-Action Models? Vision-language-action models can go further by generating robot actions directly from visual observations and language instructions. These models are particularly promising for dexterous tasks that are difficult to describe using conventional planning alone. However, the complementary architecture remains relevant even when a VLA model produces low-level actions. MATLAB and Simulink can be used to simulate the resulting behavior, monitor robot states, check operating constraints, evaluate performance across scenarios, and compare learned policies with established planning and control approaches. A conventional planner or controller can also serve as an alternative execution path when the learned policy produces low-confidence, infeasible, or undesirable behavior. This allows development teams to choose the appropriate balance between learned behavior and algorithmic execution for each part of the application. A Model-Agnostic Engineering Architecture Gemini Robotics-ER provides a clear example of high-level embodied reasoning. NVIDIA Cosmos 3 expands the broader Physical AI landscape with models for reasoning, world generation, simulation, and action development. The larger opportunity is not limited to one model provider. A model-agnostic robotics architecture can connect external reasoning or action models to a consistent MATLAB and Simulink engineering workflow. Development teams can evaluate different foundation models while preserving the robot models, motion-planning algorithms, controllers, simulation scenarios, tests, and hardware interfaces they have already developed. This separation offers several advantages. The reasoning model can evolve without requiring the entire robotics stack to be rebuilt. Robotics algorithms can be tested independently of the model’s semantic reasoning. Engineers can compare multiple models using the same scenes and performance metrics. Simulation can expose the system to edge cases before hardware testing. The validated execution components can then be reused across applications and deployment targets. Combining Adaptability with Engineering Rigor Embodied reasoning models can make robots easier to instruct and more adaptable to new objects, tasks, and environments. Robotics System Toolbox provides the geometric, algorithmic, and execution capabilities needed to turn those intentions into robot motion. Simulink provides the system-level environment for integrating, simulating, testing, and deploying the complete closed-loop application. The complementary story is therefore not foundation models versus classical robotics. It is: Foundation models for perception, semantic understanding, and task reasoning Robotics algorithms for feasible motion, trajectory generation, and control Simulation and testing for system-level evaluation Closed-loop feedback for verification and replanning Deployment workflows for moving from prototypes to physical systems Embodied reasoning can give a robot a better understanding of what needs to be done. MATLAB and Simulink help transform that understanding into actions that are executable, testable, repeatable, and deployable. That combination provides a practical path from Generative AI to trustworthy Physical AI. Add this as the final paragraph: Embodied reasoning helps a robot understand what should happen next. MATLAB and Simulink help ensure that it can happen reliably in the physical world. If you are interested in exploring this workflow or applying it to your robotics application, please reach out to us. create alt text for this image Diagram of a closed-loop robotics workflow. A natural-language instruction—“Move elbow fittings to left table”—and a camera observation image are sent to an external embodied reasoning model, such as Gemini Robotics-ER or NVIDIA Cosmos 3. The model performs task reasoning and verification, then passes commands to MATLAB and Simulink robotics engineering tools, including CHOMP motion planning and inverse kinematics. These tools generate executable actions for the robot, while the camera continuously observes the robot and environment to provide feedback to the reasoning model. in short Embodied reasoning interprets instructions and camera input, then uses MATLAB robotics tools for motion planning, control, and closed-loop task verification. write alt text Simulink workflow integrating Gemini Robotics for task reasoning and verification, Robotics System Toolbox for motion planning and control, and simulation or robot hardware for execution with camera, depth, and task feedback.

4 Why Keep Planning and Control in Engineering Tools?

A reasoning model may correctly identify that an elbow fitting belongs on the left table. But semantic correctness does not guarantee feasible robot motion.

The requested target could be:

  • Outside the robot’s workspace
  • Near a joint limit
  • In collision with the bin or table
  • Incompatible with the selected gripper orientation
  • Reachable only through an undesirable configuration

Robotics System Toolbox can evaluate the requested action against explicit robot and environment models. Inverse kinematics can identify valid configurations. Collision checking can reject unsafe paths. Motion planners can find feasible alternatives, and trajectory-generation algorithms can convert the path into smooth robot motion.

The reasoning model does not need to relearn these established capabilities. Instead, it can use them as trusted engineering tools.

 

5 Closing the Loop with Task Verification

One of the most valuable roles of embodied reasoning is not only planning the task, but also verifying the result – task verification.

A grasp may fail. An object may move unexpectedly. The destination may become occupied. The robot may place the wrong object or stop before completing the full instruction.

After each action, new observations can be returned to the reasoning model so it can determine:

  • Was the intended object grasped?
  • Was it placed at the correct destination?
  • Are there more objects to process?
  • Has the scene changed?
  • Should the task be replanned?

 

At the same time, MATLAB and Simulink can monitor execution-level conditions such as trajectory validity, joint behavior, collisions, and controller performance.

This creates two complementary verification loops:

  • Task-level verification: Did the robot accomplish the user’s intent?
  • Execution-level verification: Was the motion feasible and performed within the system constraints?

Together, they provide a more robust workflow than reasoning or conventional robot programming alone.

 

6 Maintaining Flexibility Without Rebuilding the Robotics Stack

This architecture can also be model-agnostic.

Different embodied reasoning models may evolve rapidly. One model may provide better spatial reasoning, another may provide faster inference, and another may be better suited for a particular robot or industrial environment.

By separating reasoning from execution, the external model can change while the engineering workflow remains largely intact.

The same MATLAB and Simulink components can continue to provide:

  • Robot models
  • Coordinate systems
  • Collision environments
  • Motion planners
  • Controllers
  • Simulation scenarios
  • Test cases
  • Hardware interfaces
  • Deployment workflows

 

This allows engineers to compare external reasoning models using a consistent robotics environment rather than rebuilding the complete system for every model.

It also makes it possible to test the two layers independently. The reasoning model can be evaluated on object selection, task decomposition, spatial understanding, and completion detection. The execution layer can be evaluated on reachability, planning success, collision avoidance, trajectory quality, and controller performance.

 

 

7 How This Differs from the VLA Approach

The VLA approach described in the first article combines perception, planning, and robot action within a learned model.

The embodied reasoning architecture separates high-level intelligence from robot execution. In this architecture, the VLM-based model interprets the scene and task, while the robotics engineering tools generate and execute feasible robot motion.

End-to-end VLA VLM plus Engineering Tools
. Model generates robot actions . Model generates task goals and decisions
. Perception and action are learned together . Reasoning and execution are separated
. Well suited for learned, adaptive behavior . Well suited for constraint-aware workflows
. Behavior depends heavily on training . Execution uses explicit models and algorithms
. Full policy is evaluated as one system . Reasoning and execution can be tested separately

 

The two approaches can also be combined. An embodied reasoning model could select a task and determine the task sequence. A VLA policy could execute a dexterous subtask such as grasping a deformable object. A conventional planner could handle free-space robot motion. A feedback controller could execute the final trajectory.

The right architecture does not have to be entirely learned or entirely model-based.

 

8 The Role of MATLAB and Simulink

The value of MATLAB and Simulink in this workflow is not to compete with large external foundation models.

Their role is to connect those models with the engineering processes required to build working robotic systems.

That includes:

  • Converting semantic task outputs into robot goals
  • Evaluating feasibility using robot models
  • Planning and controlling robot motion
  • Simulating the complete closed-loop workflow
  • Testing normal and failure scenarios
  • Comparing learned and algorithmic approaches
  • Integrating sensors, algorithms, and hardware
  • Moving from desktop prototypes to deployment targets

Embodied reasoning provides adaptability at the task level. MATLAB and Simulink provide structure, analysis, and repeatability at the execution level.

 

9 From Generative AI to Physical AI

The first wave of generative AI for robotics demonstrated the potential of end-to-end models that connect language, vision, and action.

The next step is integrating these capabilities into complete engineering systems.

External embodied reasoning models can provide adaptability, semantic understanding, and task verification. MATLAB and Simulink can provide the algorithms and engineering workflows required to convert those decisions into feasible, testable, and deployable robot actions.

The story is therefore not foundation models versus classical robotics.

It is a complementary architecture:

  • Embodied reasoning for understanding the scene and task
  • Task verification for monitoring progress
  • Robotics algorithms for planning and control
  • Simulation for integrated system evaluation
  • Model-Based Design for testing and deployment

Embodied reasoning helps a robot understand what should happen next.

MATLAB and Simulink can help turn that understanding into robot actions that are feasible, testable, repeatable, and deployable.

 

That complementary architecture provides a practical path from generative AI reasoning to reliable action in the physical world. If you are interested in exploring this workflow or applying it to your robotics application, please reach out to us.

|
Category: 
AI in Robotics
  • print

Comments

To leave a comment, please click here to sign in to your MathWorks Account or create a new one.