Using MATLAB to Detect Cookies
Today, I'd like to introduce a guest blogger, Tom Lowell, who is a program manager here at MathWorks. He works in our hardware connectivity group, and plays with Arduinos and Raspberry Pis in his spare time. He's relatively new to both MathWorks and MATLAB and decided to write a cookie detector as his first MATLAB project.
Contents
Motivation
When I joined MathWorks, one of the first traditions I learned was that a large plate of home made cookies is delivered every Friday to every floor of every building here in our corporate headquarters in Natick. The problem is, they go fast! And you don't know when they'll be delivered. So if you're in a meeting or hunkered down at your desk, it's easy to miss out.
This seemed like a great opportunity to get some hands-on experience with MATLAB's image processing capabilities and solve an important problem. I'll describe the cookie detector and how it's built. And hopefully this will whet your appetite to go off and explore MATLAB's image processing capabilities on your own.
Setup
The cookie detector does one thing. It detects when a large circular plate of circular cookies is placed anywhere on the counter in the kitchen area on my floor. The system is very simple with only two components; an IP camera in the kitchen and MATLAB running on my computer.
IP cameras are similar to web cams except they run independently from any particular computer. They're attached to your network just like your laptop, your printer, or your smart phone. Most IP cameras are accessible via their IP address. Many, including mine, have a tiny web server running on their chipset. You just type their IP address into your web browser and you get back a JPG image or even video. The IP camera that I happen to be using is a Foscam FI8910W. But you can use any IP camera that supports URL addressing. Your MATLAB code will need to know the IP address of your IP camera. Therefore, it's desirable to configure your IP camera with a static IP address so it doesn't change over time.
You need a nearby outlet to power the IP camera, and a network connection. Most IP cameras support both WIFI or hardwired connections. We're using a WIFI connection because the computer that's running MATLAB is on the other side of the building. The ceiling in our kitchen happens to be metal. This made mounting the camera a breeze (using magnets).
Code
All code for this project is contained within MATLAB. You will you also need the Image Processing Toolbox. The code is very simple. The main function loops forever, until a plate of cookies is detected. It downloads a new image from the camera every 30 seconds. MATLAB analyses the image using its built-in function imfindcircles. Once the cookies are detected, it sends a text message to several members on our team and then exits.
Getting the image from the camera into MATLAB was is the easiest part of the project. The camera has a tiny, built-in web server running on it that will return an image when requested via a URL. And MATLAB has several functions that will fetch data from web servers. I used webread which is new in R2014b. It reads content from a RESTful web service and returns it as Internet media types such as JSON, XML, images, or text.
Create a string with the camera's IP address plus specific API commands to pass to the camera. The format of this URL is camera specific. - snapshot.cgi tells the camera that we want a single image This camera also requires user and pwd as RESTful name-value pairs.
IPcamURL = 'http://172.31.28.24:1026/snapshot.cgi'; user = 'cookies'; password = 'lovecookies'
Use a timeout so that network issues don't cause it to hang.
options = weboptions('Timeout', 10);The IP camera's web interface returns content as a JPG image stream. webread converts that for you into an RGB image that is immediately usable by MATLAB functions.
RGB = webread(IPcamURL, 'user', user, 'pwd', password, options); imshow(RGB);
The analysis for cookies is also pretty straightforward. The code first checks the image for a large circle (the round cookie plate) and then checks the image for smaller circles (cookies) contained within the large circle. My code calls the Image Processing Toolbox function imfindcircles, passing it the image from the camera, along with a range of valid radii to check for. Any results are returned in a vector of circles.
Check for a plate. If one or more plates are found, check for cookies... detectPlateImfind() and detectCookiesImfind() are just customized wrappers around imfindcircles
[plateCenter, plateRadius] = detectPlateImfind(tempJpgFile, smallPlateRad, bigPlateRad, 'g', 1);
nCookies = 0;
nPlates = size(plateRadius, 1);
if nPlates > 0
nCookies = detectCookiesImfind(tempJpgFile, smallCookieRad, bigCookieRad, 'r', 0, plateCenter, plateRadius);
endThe analysis takes about 1 second per image on my MacBook Air.
Tuning
Writing the basic code was easy because MATLAB insulates the programmer from the complexities of the various image-processing algorithms. However, you can (and need to) tune your code by passing various parameters to these image processing algorithms. I'll briefly describe the type of tuning I did here. For more information, there's a great example of tuning imfindcircles in the MathWorks documentation center.
First, there are few things to notice about the real life image above. The plate is a perfect circle with well-defined edges, a known radius, and high contrast relative to the counter top. Also, the cookies are generally circular but they're a bit irregular. They may be different colors and sizes each week.
The eventual call to the built-in function imfindcircles takes advantage of all of these. Here's the call to find the plate. The range of acceptable plate radii is passed in (smallest, biggest). The "object polarity" of the silver plate relative to the dark blue counter is passed in as "Bright". And some other parameters are set that were determined by tuning with a positive test case.
Find the plate(s)
[cirCen, cirRad, cirMetric] = imfindcircles(grayimg,[smallest, biggest],... 'ObjectPolarity','Bright', 'Method', 'Twostage','Sensitivity',0.96);
Notice that the parameters to the second call to imfindcircles to find the cookies are different. The object polarity is "dark" because the cookies are dark relative to the silver plate. A different (default) detection method was used and an edge threshold was used.
Find any cookies (on the plates)
[cirCen, cirRad] = imfindcircles(grayimg,[smallest, biggest],...
'ObjectPolarity','dark','Sensitivity',0.96, 'EdgeThreshold',0.05);There are trade-offs to be made when you adjust these parameters. For example, if you want to detect every cookie on the plate, you might have to relax the parameters so much that the algorithm also incorrectly identifies other artifacts in your image as circles (other shapes, arcs, flashes of light). But if you tighten up your parameters to filter out those incorrect detections, you may lose detection of overlapping cookies, irregularly shaped cookies, etc.
The same logic applies to the plate or anything else you might want to detect. We don't want to miss the cookie plate, but we also don't want the tuning to be so permissive that we accidentally send 20 people rushing to the kitchen when someone sets their lunch plate on the counter.
The take-way here is that there is no one set of parameters to imfindcircles or any other image processing algorithm that will guarantee perfect results. You need to tune the parameters based on the real world characteristics of your image (such a the shape and color of the object, the lighting, angle, and other factors). This tuning is iterative and can take a while.
The Output
The output is a display of the image with the plate circled in green and the cookies in red. This is very easy to produce with MATLAB's basic plotting functions because imfindcircles returns a vector of circles (including the center points on the image and radii).
Although this is a pretty simple image, there are a few things to notice. It correctly detected plates of two different sizes. However, there are some false detects and it didn't get the exact number of cookies right. Lastly, those red crosses seen outside of the plate boundaries are detections that the loosely tuned algorithm thought were circles. However, some post-processing code was able to filter those out by determining they were not on the plate. This code can detect the plates with very good accuracy because of their well-defined shape and contrast. The code does not do as well with the cookies, which are irregular and offer less contrast relative to their background.
Remember, the object here is not a perfect cookie detector. The goal is correct and timely notification of the cookies' arrival. And it's pretty good at that. These cookies were detected before they were unwrapped!
Notification
Once the code has determined that there is a plate of cookies, it sends a text message to interested team members. I'm not going to cover the text messaging in this blog. It works by sending an email through the unique mail gateway of the cellular carriers. For example, to send a text to a Verizon user, MATLAB can send an email to 'xxxxxxxxxx@vtext.com' (where 'xxxxxxxxxx' is the cell phone number). I used Ke Feng's send text message example that's posted on MATLAB Central. The cool thing is that it's all done in MATLAB.
Do You Have a Project that Uses an IP camera?
Do you have a project that uses an IP camera? We'd love to hear how you might use MATLAB with your IP camera project at school, work, or home. Tell us about it here.
