Today's post is by guest blogger Isaac Bruss. Isaac has been a course developer for MathWorks since 2019. His PhD is in computational physics from UMass Amherst. Isaac has helped launch and support several courses on Coursera, on topics such as robotics, machine learning, and computer vision. Isaac's post is based on a lesson from the <\/span>Image Processing for Engineering and Science Specialization on Coursera<\/span><\/a>.<\/span><\/div>Introduction<\/span><\/h2>In this post, I\u2019ll analyze a video of a container filling with liquid. Now, that might not sound like the most exciting thing in the world, but imagine, for example, you're a quality control engineer responsible for ensuring that the rate of filling bottles is consistent. You need to determine if this rate is constant or varies over time. To answer this question, you\u2019d need to process each individual frame from the video file. But where would you start?<\/span><\/div>Well, in this case, I started out with the <\/span>Video Viewer App<\/span><\/a> to become more familiar with the video. This app allows you to see the video's size, frame rate, and total number of frames along the bottom bar. By using this app, you can quickly get a sense of what I\u2019m working with by playing the video and navigating between frames.<\/span><\/div><\/img><\/div>Creating the Segmentation Function<\/span><\/h2>With the goal of measuring the liquid within the container, I needed a segmentation function that can separate the liquid from the background. To ensure I develop a robust algorithm, I export a few representative frames to create and test a segmentation function.<\/span><\/div><\/span><\/div><\/img><\/div><\/span><\/div>The <\/span>Color Thresholder App<\/span><\/a> is a powerful tool for interactively segmenting images based on color. This app is particularly useful for images that have distinct color differences between the object of interest and the background. In this case, I used the region selection tool to highlight pixels of interest within the L*a*b* color space. This allowed me to isolate the dark liquid from both the purple background and white foam at the top. <\/span><\/div><\/img><\/div><\/span><\/div>Once I'm satisfied with my thresholding results, I export this segmentation function from the app as a \u201c.m\u201d file named \u201cliquidMask\u201d.<\/span><\/div>Check the Segmentation Function<\/span><\/h2>This custom function was <\/span>developed using only the few frames I exported from the video. But to be confident in the results, I need to ensure that it accurately isolates the dark liquid in all the frames. To do this, I first create a <\/span>video reader object<\/span><\/a>.<\/span><\/div>v = VideoReader(<\/span>\"liquidVideo.mp4\"<\/span>)\t<\/span><\/span><\/div>v = <\/span><\/div>  VideoReader with properties:\r\n\r\n   General Properties:\r\n            Name: 'liquidVideo.mp4'\r\n            Path: 'C:\\MATLAB Drive\\Play\\Blog Posts\\video analysis'\r\n        Duration: 24\r\n     CurrentTime: 0.1000\r\n       NumFrames: <Calculating...> learn more\r\n\r\n   Video Properties:\r\n           Width: 120\r\n          Height: 216\r\n       FrameRate: 10\r\n    BitsPerPixel: 24\r\n     VideoFormat: 'RGB24'\r\n\r\n<\/div><\/div><\/div><\/div><\/div><\/div>Then I loop through the frames using the <\/span>hasFrame<\/span><\/a> function. Inside the loop, each frame is passed into the custom liquidMask function, and the resulting mask is displayed side-by-side using the <\/span>montage<\/span><\/a> function.<\/span><\/div>while <\/span>hasFrame(v) <\/span>% Loop through all frames<\/span><\/span><\/div><\/div>    img = readFrame(v); <\/span>% Read a frame<\/span><\/span><\/div><\/div>    bw = liquidMask(img); <\/span>% Apply the custom segmentation function<\/span><\/span><\/div><\/div>    montage({img,bw},<\/span>'BorderSize'<\/span>,[20,20],<\/span>'BackgroundColor'<\/span>,[0.5,0.5,0.5]);<\/span><\/span><\/div><\/div>    drawnow<\/span><\/span><\/div><\/div>end<\/span><\/span><\/div><\/div><\/div><\/img><\/div><\/span><\/div>The resulting segmented video frames look great! Further morphological operations to better segment the region of interest are possible, but to get a quick example up and running they aren\u2019t needed. <\/span><\/div>Analyzing Liquid Percentage<\/span><\/h2>Now I can move on to the final part: calculating the percentage of the container that is filled with water at each frame.<\/span><\/div>To get started, I need to rewind the video to the beginning using the CurrentTime property of the VideoReader.<\/span><\/div>v.CurrentTime = 0;\t<\/span><\/span><\/div><\/div><\/div>Next, I initialize two variables to store values for each frame: the percentage of the container filled, and the time stamp.<\/span><\/div>nFrames = v.NumFrames;<\/span><\/span><\/div><\/div>percents = zeros(nFrames,1);<\/span><\/span><\/div><\/div>times = zeros(nFrames,1);\t<\/span><\/span><\/div><\/div><\/div>Now I\u2019m ready to use a for-loop to read through the video, segment each image, and calculate the percentage of the container filled at each frame. The percentage calculation is done by counting the number of pixels that are filled with liquid and dividing it by the total number of pixels.<\/span><\/div>for <\/span>i = 1:nFrames <\/span>% Loop through all frames<\/span><\/span><\/div><\/div>    img = readFrame(v); <\/span>% Read a frame<\/span><\/span><\/div><\/div>    bw = liquidMask(img); <\/span>% Create the binary mask using the custom function<\/span><\/span><\/div><\/div>    <\/span><\/span><\/div><\/div>    liquidPart = nnz(bw); <\/span>% Calculate the number of liquid (true) pixels<\/span><\/span><\/div><\/div>    perc = liquidPart\/numel(bw); <\/span>% Calculate percentage filled with liquid<\/span><\/span><\/div><\/div>    <\/span><\/span><\/div><\/div>    time = v.CurrentTime; <\/span>% Mark the current time<\/span><\/span><\/div><\/div>    <\/span><\/span><\/div><\/div>    percents(i) = perc; <\/span>% Save the fill-percentage value<\/span><\/span><\/div><\/div>    times(i) = time; <\/span>% Save the time stamp<\/span><\/span><\/div><\/div>end<\/span><\/span><\/div><\/div><\/div>Now that the data has been gathered, I plot the percentage of the container filled versus time to see how quickly the liquid fills the container.<\/span><\/div>figure<\/span><\/span><\/div><\/div>plot(times,percents*100)<\/span><\/span><\/div><\/div>title(<\/span>'Percentage of Container Filled with Liquid vs Time'<\/span>)<\/span><\/span><\/div><\/div>xlabel(<\/span>'Time (seconds)'<\/span>)<\/span><\/span><\/div><\/div>ylabel(<\/span>'Percentage Filled'<\/span>)<\/span><\/span><\/div><\/div><\/div><\/div><\/div><\/div><\/div><\/div><\/div>Look at that! The fill rate is not consistent. From the plot, I see that the flow slowed down around 8 seconds, increased at 15 seconds, and slowed again at 20 seconds. <\/span><\/div>This post is based on a lesson from the <\/span>Image Processing for Engineering and Science Specialization on Coursera<\/span><\/a>. There you\u2019ll be able to complete many projects like this one, from segmenting cracks in images of concrete to counting cars in a video of traffic.<\/span><\/div>___<\/span><\/div>Thanks, Isaac!<\/span><\/div><\/span><\/div><\/span><\/div><\/span><\/div><\/div>\r\n