{"id":10259,"date":"2023-09-11T06:00:55","date_gmt":"2023-09-11T10:00:55","guid":{"rendered":"https:\/\/blogs.mathworks.com\/student-lounge\/?p=10259"},"modified":"2023-09-10T20:33:42","modified_gmt":"2023-09-11T00:33:42","slug":"using-ensemble-learning-to-create-accurate-machine-learning-algorithms","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/student-lounge\/2023\/09\/11\/using-ensemble-learning-to-create-accurate-machine-learning-algorithms\/","title":{"rendered":"Using Ensemble Learning to Create Accurate Machine Learning Algorithms"},"content":{"rendered":"
\n
In today’s post, <\/span>Grace<\/span><\/a> from the Student Programs Team will show how you can started with ensemble learning. Over to you, Grace!<\/span><\/div>\n
When building a predictive machine learning model, there are many ways to improve it’s performance: try out different algorithms, optimize the parameters of the algorithm, find the best way to divide and process your data, and more. Another great way to create accurate predictive models is through ensemble learning.<\/span><\/div>\n

What is ensemble learning?<\/h2>\n
Ensemble learning is the practice of combining multiple machine learning models into one predictive model. Some types of machine learning algorithms are considered weak learners, <\/span>meaning that they are highly sensitive to the data that is used to train them and are prone to inaccuracies. Creating an ensemble<\/span> of weak learners and aggregating their results to make predictions on new observations often results in a single higher-quality model. At it’s simplest, ensemble learning can be represented with the animation below:<\/div>\n
\"EnsembleGif.gif\"<\/div>\n
Ensemble learning can be used for a wide variety of machine and deep learning methods. Today, I will show how to create an ensemble of machine learning models for a regression problem, though the workflow will be similar for classification problems as well. Let’s get started!<\/div>\n

1. Prepare the data<\/h2>\n
For this problem, we have a set of tabular data pertaining to cars, as shown below:<\/div>\n
\n
\n
load carbig<\/span><\/span><\/div>\n<\/div>\n
\n
carTable = table(Acceleration,Cylinders,Displacement, …<\/span><\/span><\/div>\n<\/div>\n
\n
Horsepower,Model_Year,Weight,MPG);<\/span><\/div>\n<\/div>\n
\n
head(carTable)<\/span><\/div>\n
\n
\n
Acceleration<\/strong> Cylinders<\/strong> Displacement<\/strong> Horsepower<\/strong> Model_Year<\/strong> Weight<\/strong> MPG<\/strong>
\n____________<\/strong> _________<\/strong> ____________<\/strong> __________<\/strong> __________<\/strong> ______<\/strong> ___<\/strong>12 8 307 130 70 3504 18
\n11.5 8 350 165 70 3693 15
\n11 8 318 150 70 3436 18
\n12 8 304 150 70 3433 16
\n10.5 8 302 140 70 3449 17
\n10 8 429 198 70 4341 15
\n9 8 454 220 70 4354 14
\n8.5 8 440 215 70 4312 14<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
Our goal is to create a model that can accurately predict what a car’s mileage per gallon (MPG) will be. With any data problem, you should take the time to explore and preprocess the data, but for this tutorial I will just do some simple steps. For more information on cleaning your data, check out this example<\/a> that shows a lot of great ways you can preprocess tabular data!<\/div>\n
First, I’ll check if our set has any rows with missing data, as this will inform some of our decisions later.<\/div>\n
\n
\n
missingElements = ismissing(carTable);<\/span><\/div>\n<\/div>\n
\n
rowsWithMissingValues = any(missingElements,2);<\/span><\/div>\n<\/div>\n
\n
missingValuesTable = carTable(rowsWithMissingValues,:)<\/span><\/div>\n
\n
\n
\n
missingValuesTable = 14\u00d77 table <\/span><\/div>\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
<\/th>\nAcceleration<\/th>\nCylinders<\/th>\nDisplacement<\/th>\nHorsepower<\/th>\nModel_Year<\/th>\nWeight<\/th>\nMPG<\/th>\n<\/tr>\n<\/thead>\n
1<\/th>\n17.5000<\/td>\n4<\/td>\n133<\/td>\n115<\/td>\n70<\/td>\n3090<\/td>\nNaN<\/td>\n<\/tr>\n
2<\/th>\n11.5000<\/td>\n8<\/td>\n350<\/td>\n165<\/td>\n70<\/td>\n4142<\/td>\nNaN<\/td>\n<\/tr>\n
3<\/th>\n11<\/td>\n8<\/td>\n351<\/td>\n153<\/td>\n70<\/td>\n4034<\/td>\nNaN<\/td>\n<\/tr>\n
4<\/th>\n10.5000<\/td>\n8<\/td>\n383<\/td>\n175<\/td>\n70<\/td>\n4166<\/td>\nNaN<\/td>\n<\/tr>\n
5<\/th>\n11<\/td>\n8<\/td>\n360<\/td>\n175<\/td>\n70<\/td>\n3850<\/td>\nNaN<\/td>\n<\/tr>\n
6<\/th>\n8<\/td>\n8<\/td>\n302<\/td>\n140<\/td>\n70<\/td>\n3353<\/td>\nNaN<\/td>\n<\/tr>\n
7<\/th>\n19<\/td>\n4<\/td>\n98<\/td>\nNaN<\/td>\n71<\/td>\n2046<\/td>\n25<\/td>\n<\/tr>\n
8<\/th>\n20<\/td>\n4<\/td>\n97<\/td>\n48<\/td>\n71<\/td>\n1978<\/td>\nNaN<\/td>\n<\/tr>\n
9<\/th>\n17<\/td>\n6<\/td>\n200<\/td>\nNaN<\/td>\n74<\/td>\n2875<\/td>\n21<\/td>\n<\/tr>\n
10<\/th>\n17.3000<\/td>\n4<\/td>\n85<\/td>\nNaN<\/td>\n80<\/td>\n1835<\/td>\n40.9000<\/td>\n<\/tr>\n
11<\/th>\n14.3000<\/td>\n4<\/td>\n140<\/td>\nNaN<\/td>\n80<\/td>\n2905<\/td>\n23.6000<\/td>\n<\/tr>\n
12<\/th>\n15.8000<\/td>\n4<\/td>\n100<\/td>\nNaN<\/td>\n81<\/td>\n2320<\/td>\n34.5000<\/td>\n<\/tr>\n
13<\/th>\n15.4000<\/td>\n4<\/td>\n121<\/td>\n110<\/td>\n81<\/td>\n2800<\/td>\nNaN<\/td>\n<\/tr>\n
14<\/th>\n20.5000<\/td>\n4<\/td>\n151<\/td>\nNaN<\/td>\n82<\/td>\n3035<\/td>\n23<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
There are a total of 14 rows with missing data, 8 of which are missing the ‘MPG’ value. I’ll remove these rows, as they are not helpful for training, but we will use the other rows as they could still provide helpful information when training.<\/div>\n
\n
\n
rowsMissingMPG = ismissing(carTable.MPG);<\/span><\/div>\n<\/div>\n
\n
carTable(rowsMissingMPG,: ) = []<\/span><\/div>\n
\n
\n
\n
carTable = 398\u00d77 table <\/span><\/div>\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
<\/th>\nAcceleration<\/th>\nCylinders<\/th>\nDisplacement<\/th>\nHorsepower<\/th>\nModel_Year<\/th>\nWeight<\/th>\nMPG<\/th>\n<\/tr>\n<\/thead>\n
1<\/th>\n12<\/td>\n8<\/td>\n307<\/td>\n130<\/td>\n70<\/td>\n3504<\/td>\n18<\/td>\n<\/tr>\n
2<\/th>\n11.5000<\/td>\n8<\/td>\n350<\/td>\n165<\/td>\n70<\/td>\n3693<\/td>\n15<\/td>\n<\/tr>\n
3<\/th>\n11<\/td>\n8<\/td>\n318<\/td>\n150<\/td>\n70<\/td>\n3436<\/td>\n18<\/td>\n<\/tr>\n
4<\/th>\n12<\/td>\n8<\/td>\n304<\/td>\n150<\/td>\n70<\/td>\n3433<\/td>\n16<\/td>\n<\/tr>\n
5<\/th>\n10.5000<\/td>\n8<\/td>\n302<\/td>\n140<\/td>\n70<\/td>\n3449<\/td>\n17<\/td>\n<\/tr>\n
6<\/th>\n10<\/td>\n8<\/td>\n429<\/td>\n198<\/td>\n70<\/td>\n4341<\/td>\n15<\/td>\n<\/tr>\n
7<\/th>\n9<\/td>\n8<\/td>\n454<\/td>\n220<\/td>\n70<\/td>\n4354<\/td>\n14<\/td>\n<\/tr>\n
8<\/th>\n8.5000<\/td>\n8<\/td>\n440<\/td>\n215<\/td>\n70<\/td>\n4312<\/td>\n14<\/td>\n<\/tr>\n
9<\/th>\n10<\/td>\n8<\/td>\n455<\/td>\n225<\/td>\n70<\/td>\n4425<\/td>\n14<\/td>\n<\/tr>\n
10<\/th>\n8.5000<\/td>\n8<\/td>\n390<\/td>\n190<\/td>\n70<\/td>\n3850<\/td>\n15<\/td>\n<\/tr>\n
11<\/th>\n10<\/td>\n8<\/td>\n383<\/td>\n170<\/td>\n70<\/td>\n3563<\/td>\n15<\/td>\n<\/tr>\n
12<\/th>\n8<\/td>\n8<\/td>\n340<\/td>\n160<\/td>\n70<\/td>\n3609<\/td>\n14<\/td>\n<\/tr>\n
13<\/th>\n9.5000<\/td>\n8<\/td>\n400<\/td>\n150<\/td>\n70<\/td>\n3761<\/td>\n15<\/td>\n<\/tr>\n
14<\/th>\n10<\/td>\n8<\/td>\n455<\/td>\n225<\/td>\n70<\/td>\n3086<\/td>\n14<\/td>\n<\/tr>\n
\u22ee<\/th>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n<\/div>\n
<\/div>\n
<\/div>\n
<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
Last, I will split our data into a training set and a testing set, which will be used to teach and evaluate the ensemble, respectively. Using the dividerand<\/span><\/a> function, I put 70% of the data into the training set and 30% into the testing set as a starting point, but you can test out different divisions of data when building your own models.<\/div>\n
\n
\n
numRows = size(carTable,1);<\/span><\/div>\n<\/div>\n
\n
[trainInd, ~, testInd] = dividerand(numRows, .7, 0, .3);<\/span><\/div>\n<\/div>\n
\n
<\/div>\n<\/div>\n
\n
trainingData = carTable(trainInd, :);<\/span><\/div>\n<\/div>\n
\n
testingData = carTable(testInd, :);<\/span><\/div>\n<\/div>\n<\/div>\n

2. Create an ensemble<\/h2>\n
Now that our data is ready, it’s time to start creating the ensemble! I’ll start by showing the easiest way to create an ensemble using the default parameters for each individual learner, and then I’ll also show how to use templates to customize your weak learners.<\/div>\n

Using Bulit-In Algorithms<\/h3>\n
You can create an ensemble for regression by using fitrensemble<\/span><\/a> (<\/span>fitcensemble<\/span><\/a> for classification). With just this function and your data, you could have an ensemble ready by executing the following line of code:<\/div>\n
\n
Mdl = fitrensemble(trainingData, ‘MPG’<\/span>);<\/span><\/div>\n<\/div>\n
This will use all the default settings for training an ensemble of weak regression learners: 100 trees are trained and they are aggregated using the least-squares boosting (LSBoost) algorithm.<\/div>\n
However, fitrensemble<\/span> also provides the option to customize the ensemble settings, so I will specify a few of these settings. First, I want to use the ‘Bag’ method of aggregation instead of the ‘LSBoost’ method because it tends to have higher accuracy and our dataset is relatively small. For a full list of aggregation algorithms and some suggestions on how to choose a starting algorithm, check out this documentation page<\/a>!<\/div>\n
I also want to specify how many learners will be in the ensemble, which is set by the ‘NumLearningCycles’ property. To choose how many learners the ensemble will have, try starting with several dozen, training the ensemble, and then checking the ensemble quality. If the ensemble is not accurate and could benefit from more learners, then you can adjust this number or add them in later! For now, I’ll start with 30 learners.<\/div>\n
Both of these options are set using Name-Value arguments, as shown below.<\/div>\n
\n
\n
Mdl = fitrensemble(trainingData, ‘MPG’<\/span>, ‘Method’<\/span>, ‘Bag’<\/span>, ‘NumLearningCycles’<\/span>, 30);<\/span><\/div>\n<\/div>\n<\/div>\n
And just like that, we’ve trained an ensemble of 50 learners that are ready to be used on new data!<\/div>\n

Using Templates<\/h3>\n
There may be times when you want to change some parameters of the individual learners, not just of the ensemble. To do that, we can use learner templates.<\/span><\/div>\n
Unless otherwise specified, fitrensemble<\/span> creates an ensemble of default regression tree learners, but this may not always be what you want. As we saw earlier, our data has some missing values, which can decrease the performance of these trees. Trees that use surrogate splits<\/a> tend to perform better with missing data than trees that don’t, so I will use the templateTree<\/span><\/a> function to specify that I want the learners to use surrogate splits.<\/div>\n
\n
\n
templ = templateTree(‘Surrogate’<\/span>,‘all’<\/span>, ‘Type’<\/span>, ‘regression’<\/span>);<\/span><\/div>\n<\/div>\n
\n
templMdl = fitrensemble(trainingData, ‘MPG’<\/span>, ‘Method’<\/span>, ‘Bag’<\/span>, ‘NumLearningCycles’<\/span>, 30, ‘Learners’<\/span>, templ);<\/span><\/div>\n<\/div>\n<\/div>\n
As before, we end up with an ensemble that can be used to make predictions on new data! While you can give fitcensemble<\/span> and fitrensemble<\/span> a cell array of learner templates, the most common usage is to give just one weak learner template.<\/div>\n

3. Evaluate the Ensemble<\/h2>\n
Once you have a trained ensemble, it’s time to see how well it performs! The predictive quality of an ensemble cannot be evaluated based on its performance on training data, as the ensemble knows it too well. It’s very likely that it will perform really well on the training data, but that does not mean it will perform well on any other data. To obtain a better idea of the quality of an ensemble, you can use one of these methods:<\/div>\n