{"id":10980,"date":"2024-04-22T08:13:11","date_gmt":"2024-04-22T12:13:11","guid":{"rendered":"https:\/\/blogs.mathworks.com\/student-lounge\/?p=10980"},"modified":"2024-04-22T08:12:31","modified_gmt":"2024-04-22T12:12:31","slug":"custom-ros-messages-for-object-detection","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/student-lounge\/2024\/04\/22\/custom-ros-messages-for-object-detection\/","title":{"rendered":"Custom ROS messages for Object Detection"},"content":{"rendered":"
\n
In today’s blog, Abhishek Shankar<\/a> will show you how you can build custom ROS messages for object detection. Over to you, Abhishek..<\/em><\/div>\n
<\/div>\n
Deep learning has proven to be extremely useful in robotics, especially in perception, and the Robot Operating System (ROS) is a great framework that allows users to build independent nodes that communicate seamlessly with each other. Integrating an object detection model with ROS, however, can be tricky. The output from the model can be a single label with the associated bounding box or an array of bounding boxes and labels, which makes publishing these results over ROS challenging. This is where custom messages prove useful, as you can construct a suitable message type based on your application and in this blog, we will see how to make use of them. Shown below is an example of the results published over ROS<\/div>\n
\"thumbnail.png\"<\/div>\n

Custom ROS Messages<\/h2>\n
ROS provides a whole lot of message types required for various robotics tasks, but it is impossible to cover every use case. Custom messages allow you to design your own message for the problem you are working on, making it easy to adapt to your particular workflow. Generating a custom message in MATLAB is simple and takes only a few steps.<\/div>\n

1. Folder structure<\/h3>\n
It is good practice to have all your custom messages under a single folder, so we’ll first create a folder called ‘custom_messages’ and inside that, create another folder ‘deep_learning’ which will have messages related to deep learning. Inside ‘deep_learning’ create a folder ‘msg’ which will have the required message files. You can follow this link<\/span><\/a>\u202fto better understand the folder structure.<\/div>\n
\n
\n
mkdir(‘custom_messages\/deep_learning\/msg’<\/span>)<\/span><\/div>\n
\n
\n
\n
<\/div>\n
<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n

2. Creating messages<\/h3>\n
Typically, there will be multiple objects and their associated labels (string),<\/span>\u202fbounding boxes (1×4 array)<\/span>\u202fand scores (float32)<\/span>. To do this, we’ll create two message types:<\/div>\n
    \n
  1. Prediction<\/span> – This will have the label, bounding box and confidence (score) of the object that has the highest score<\/li>\n
  2. PredArray<\/span> – This will be an array of Prediction<\/span> types which will have the scores, labels and bounding boxes of all the objects detected<\/li>\n<\/ol>\n
    The reason for having an array of Prediction <\/span>messages is that it is simpler than using the Float32MultiArray<\/span>\u202fdatatype, that comes with ROS, to publish the bounding boxes of multiple detected objects.<\/span>\u202fCreating these messages is simple and follows the general guidelines for custom message creation in ROS. First, we will create two files and name them Prediction.msg and PredArray.msg.<\/div>\n
    \n
    \n
    fid1 = fopen(“custom_messages\/deep_learning\/msg\/Prediction.msg”<\/span>, “wt”<\/span>);<\/span><\/div>\n<\/div>\n
    \n
    fid2 = fopen(“custom_messages\/deep_learning\/msg\/PredArray.msg”<\/span>, “wt”<\/span>);<\/span><\/div>\n<\/div>\n<\/div>\n
    We’ll define Prediction.msg as shown below<\/div>\n
    \n
    string label<\/span><\/div>\n
    float32 confidence<\/span><\/div>\n
    float32[] bbox<\/span><\/div>\n<\/div>\n
    \n
    \n
    str = [“string label”<\/span>; “float32 confidence”<\/span>; “float32[] bbox”<\/span>];<\/span><\/div>\n<\/div>\n
    \n
    fprintf(fid1, “%s\\n”<\/span>, str);<\/span><\/div>\n<\/div>\n
    \n
    fclose(fid1);<\/span><\/div>\n<\/div>\n<\/div>\n
    And the PredArray.msg is defined as<\/div>\n
    \n
    uint8 count<\/span><\/div>\n
    deep_learning\/Prediction[] PredArray<\/span><\/div>\n<\/div>\n
    \n
    \n
    str2 = [“uint8 count”<\/span>; “deep_learning\/Prediction[] PredArray”<\/span>];<\/span><\/div>\n<\/div>\n
    \n
    fprintf(fid2, “%s\\n”<\/span>, str2);<\/span><\/div>\n<\/div>\n
    \n
    fclose(fid2);<\/span><\/div>\n<\/div>\n<\/div>\n
    where count is the number of objects detected in an image.<\/div>\n

    3. Building the messages<\/h3>\n
    Once the msg files are created, we’ll build them in MATLAB so that it shows up in the available list of ROS messages. To build the messages, go the ‘custom_messages’ folder level and pass in the folder name to rosgenmsg<\/span><\/a><\/div>\n
    \n
    \n
    rosgenmsg(‘custom_messages’<\/span>)<\/span><\/div>\n
    \n
    \n
    Identifying message files in folder ‘C:\/Users\/abshanka\/OneDrive – MathWorks\/Documents\/Blogs\/ROS_custom_messages\/custom_messages’.Done.
    \nValidating message files in folder ‘C:\/Users\/abshanka\/OneDrive – MathWorks\/Documents\/Blogs\/ROS_custom_messages\/custom_messages’.Done.
    \n[1\/1] Generating MATLAB interfaces for custom message packages… Done.
    \nRunning catkin build in folder ‘C:\/Users\/abshanka\/OneDrive – MathWorks\/Documents\/Blogs\/ROS_custom_messages\/custom_messages\/matlab_msg_gen_ros1\/win64’.
    \nBuild in progress. This may take several minutes…<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
    This should bulid the messages and give us steps to use them, which is to add the generated messages to MATLAB path and refresh all message class definition as shown below. To check if the message has been added to the list of ROS messages you can run rosmsg list.<\/span><\/div>\n
    \n
    \n
    %Add the generated files to matlab path<\/span><\/span><\/div>\n<\/div>\n
    \n
    addpath(‘C:\\Users\\abshanka\\OneDrive – MathWorks\\Documents\\Blogs\\ROS_custom_messages\\custom_messages\\matlab_msg_gen_ros1\\win64\\install\\m’<\/span>)<\/span><\/div>\n<\/div>\n
    \n
    savepath<\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    %Refresh message classes<\/span><\/span><\/div>\n<\/div>\n
    \n
    clear<\/span><\/div>\n<\/div>\n
    \n
    rehash toolboxcache<\/span><\/span><\/div>\n<\/div>\n<\/div>\n

    ROS Setup<\/h2>\n
    Before we go into the deep learning part, we’ll first set up our MATLAB ROS environment variables to connect to the ROS master. If you would like to know more about ROS, check out this online tutorial<\/span><\/a>.<\/div>\n
    \n
    \n
    % ROS master is set to the IP address of the machine that has the master<\/span><\/span><\/div>\n<\/div>\n
    \n
    % running and ROS_HOSTNAME is the IP address of the machine running MATLAB.<\/span><\/span><\/div>\n<\/div>\n
    \n
    setenv(“ROS_MASTER_URI”<\/span>, “http:\/\/172.31.204.32:11311”<\/span>)<\/span><\/div>\n<\/div>\n
    \n
    setenv(“ROS_HOSTNAME”<\/span>, “172.31.204.224”<\/span>)<\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    % Call rosinit to establish a connection if it is not done so already<\/span><\/span><\/div>\n<\/div>\n
    \n
    if <\/span>~ros.internal.Global.isNodeActive<\/span><\/div>\n<\/div>\n
    \n
    rosinit<\/span><\/div>\n<\/div>\n
    \n
    end<\/span><\/span><\/div>\n<\/div>\n<\/div>\n

    Model Prediction<\/h2>\n
    Now that the custom messages have been created and a ROS connection established, let’s move to the deep learning part! In this blog, we’ll use a pretrained YOLOv4 network (you can get this by installing ‘Computer Vision Toolbox Model for YOLO v4 \ufffc<\/a>‘ from the Add-Ons library) and use a test image<\/a> of a city scene. Let’s first load the network and assign it to a variable.<\/div>\n
    \n
    \n
    if <\/span>~exist(‘detector’<\/span>, ‘var’<\/span>)<\/span><\/div>\n<\/div>\n
    \n
    detector = yolov4ObjectDetector();<\/span><\/div>\n<\/div>\n
    \n
    end<\/span><\/span><\/div>\n<\/div>\n<\/div>\n
    The model expects an image of size 608×608 so we’ll load the image and resize it appropriately and have a look at it.<\/div>\n
    \n
    \n
    img = imread(“test.jpeg”<\/span>);<\/span><\/div>\n<\/div>\n
    \n
    img = imresize(img, [608, 608]);<\/span><\/div>\n<\/div>\n
    \n
    imshow(img)<\/span><\/div>\n
    \n
    \n
    <\/div>\n
    <\/div>\n
    <\/div>\n
    <\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
    The image here shows a snapshot of what looks to be a busy street with a lot happening. Now let’s see if the model can detect the various objects in the scene!<\/div>\n
    \n
    \n
    % Thresholding detections with a score >= 0.6<\/span><\/span><\/div>\n<\/div>\n
    \n
    [bboxes,scores,labels] = detect(detector, img, ‘Threshold’<\/span>, 0.6);<\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    % insertObjectAnnotations adds the detected bounding boxes and labels to<\/span><\/span><\/div>\n<\/div>\n
    \n
    % the originial image<\/span><\/span><\/div>\n<\/div>\n
    \n
    annotatedImg = insertObjectAnnotation(img, ‘rectangle’<\/span>, bboxes, labels);<\/span><\/div>\n<\/div>\n
    \n
    imshow(annotatedImg)<\/span><\/div>\n
    \n
    \n
    <\/div>\n
    <\/div>\n
    <\/div>\n
    <\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
    Great! We’re able to detect most of the objects in the scene. You can of course play with the threshold or fine tune the network to detect more objects but for now this should do fine. Now let’s send these results as ROS topics.<\/div>\n

    Publishing over ROS<\/h2>\n
    We’ll start by defining the publishers and creating placeholder messages. Let’s have the topics as best_detection <\/span>and all_detections. <\/span>We’ll publish the object with highest score or confidence on best_detection <\/span>and send all of the detected objects on all_detections<\/span><\/div>\n
    \n
    \n
    %Define publishers using the message type built above<\/span><\/span><\/div>\n<\/div>\n
    \n
    bestPub = rospublisher(“\/best_detection”<\/span>, “deep_learning\/Prediction”<\/span>, “DataFormat”<\/span>, “struct”<\/span>);<\/span><\/div>\n<\/div>\n
    \n
    allPub = rospublisher(“\/all_detections”<\/span>, “deep_learning\/PredArray”<\/span>, “DataFormat”<\/span>, “struct”<\/span>);<\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    %Create placeholder messages<\/span><\/span><\/div>\n<\/div>\n
    \n
    bestMsg = rosmessage(bestPub);<\/span><\/div>\n<\/div>\n
    \n
    allMsg = rosmessage(allPub);<\/span><\/div>\n<\/div>\n<\/div>\n
    Now all that is remaining is to populate the messages with the respective data and publish them in a loop over ROS!<\/div>\n
    \n
    \n
    % Using an infinite loop here to continously send the ros messages<\/span><\/span><\/div>\n<\/div>\n
    \n
    while<\/span>(1)<\/span><\/div>\n<\/div>\n
    \n
    % Get the object with the highest score to be sent over ‘best_detection’<\/span><\/span><\/div>\n<\/div>\n
    \n
    [~,idx] = max(scores);<\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    % Update the fields of bestMsg<\/span><\/span><\/div>\n<\/div>\n
    \n
    bestMsg.Label = char(labels(idx));<\/span><\/div>\n<\/div>\n
    \n
    bestMsg.Confidence = scores(idx);<\/span><\/div>\n<\/div>\n
    \n
    bestMsg.Bbox = single(bboxes(idx,:)’);<\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    % Get the number of detections and loop over them to update the fields<\/span><\/span><\/div>\n<\/div>\n
    \n
    % of allMsg<\/span><\/span><\/div>\n<\/div>\n
    \n
    n = length(scores);<\/span><\/div>\n<\/div>\n
    \n
    allMsg.Count = uint8(n);<\/span><\/div>\n<\/div>\n
    \n
    for <\/span>i = 1:n<\/span><\/div>\n<\/div>\n
    \n
    allMsg.PredArray_(i).Label = char(labels(i));<\/span><\/div>\n<\/div>\n
    \n
    allMsg.PredArray_(i).Confidence = scores(i);<\/span><\/div>\n<\/div>\n
    \n
    allMsg.PredArray_(i).Bbox = single(bboxes(i, :)’);<\/span><\/div>\n<\/div>\n
    \n
    end<\/span><\/span><\/div>\n<\/div>\n
    \n
    <\/div>\n<\/div>\n
    \n
    %publish the messages<\/span><\/span><\/div>\n<\/div>\n
    \n
    send(bestPub, bestMsg);<\/span><\/div>\n<\/div>\n
    \n
    send(allPub, allMsg);<\/span><\/div>\n<\/div>\n
    \n
    end<\/span><\/span><\/div>\n<\/div>\n<\/div>\n
    That should do it! Now the topics should be visible on your ROS master PC as shown below (A subset of the 17 objects detected). Note that to view these topics you must build the messages on ROS as well which can be done following this workflow<\/span>.<\/div>\n
    \"ros-msg-output.png\"<\/div>\n
    The example taken here is a simple one where a static image is used to demonstrate publishing the detection results. This can be modified to read an image from a ROS subscriber object easily with rossubscriber<\/span>\u202f<\/span>to make it more dynamic. You can find more on handling custom messages in MATLAB here<\/a> and check out our perception series<\/a> to know how you can utilize deep learning for your robot!<\/div>\n<\/div>\n