{"id":3162,"date":"2018-12-05T10:49:30","date_gmt":"2018-12-05T15:49:30","guid":{"rendered":"https:\/\/blogs.mathworks.com\/loren\/?p=3162"},"modified":"2018-12-05T10:49:30","modified_gmt":"2018-12-05T15:49:30","slug":"debugging-grouped-operations","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/loren\/2018\/12\/05\/debugging-grouped-operations\/","title":{"rendered":"Debugging Grouped Operations"},"content":{"rendered":"

Today's guest post comes from Sean de Wolski<\/a>, one of Loren's fellow Application Engineers. You might recognize him from MATLAB answers<\/a> and the pick of the week<\/a> blog!<\/i><\/p>

One of my colleagues approached me last month and asked for help debugging an error with splitapply<\/tt><\/a>. Splitapply takes group information and applies a function to each group in the data (sort of like a pivot table). Note, that everything here also applies to the lower level but more powerful function accumarray<\/tt><\/a>.<\/p>

The documentation provides numerous simple examples for what splitapply<\/tt> does so check them out if you're not familiar with it.<\/p>

Here is an anonymized version of the data set my colleague had. The first three variables are categorical identifiers and the fourth are some data associated with them.<\/p>

load('Bears.mat'<\/span>)\r\ndisp(Bears)\r\n<\/pre>
     Candy        Animal        Sports                       Bytes                 \r\n    ________    __________    ___________    ______________________________________\r\n    Cinnamon    Black         Brown             -0.8095       0.40391             2\r\n    Cinnamon    Polar         Brown             -2.9443      0.096455             3\r\n    Cinnamon    Polar         Brown              1.4384       0.13197             9\r\n    Cinnamon    Sloth         Chicago           0.32519       0.94205             1\r\n    Cinnamon    Sloth         Chicago          -0.75493       0.95613             5\r\n    Cinnamon    Sloth         Chicago            1.3703       0.57521             2\r\n    Cinnamon    Sloth         Chicago           -1.7115       0.05978            10\r\n    Cinnamon    Sloth         Chicago          -0.10224       0.23478             8\r\n    Cinnamon    Sun           Baylor           -0.24145       0.35316             6\r\n    Cinnamon    Sun           Baylor            0.31921       0.82119             5\r\n    Cinnamon    Polar         Brown             0.31286      0.015403             1\r\n    Cinnamon    Polar         Brown            -0.86488      0.043024             7\r\n    Cinnamon    Black         Brown           -0.030051       0.16899             1\r\n    Cinnamon    Black         Brown            -0.16488       0.64912             1\r\n    Cinnamon    Polar         Brown             0.62771       0.73172             6\r\n    Cinnamon    Polar         Brown              1.0933       0.64775             1\r\n    Cinnamon    Spectacled    Coast Guard        1.1093       0.45092             9\r\n    Cinnamon    Sloth         Chicago          -0.86365       0.54701             9\r\n    Gummy       Sun           Baylor           0.077359       0.29632             8\r\n    Gummy       Sun           Baylor            -1.2141       0.74469             2\r\n    Gummy       Sun           Baylor            -1.1135       0.18896             7\r\n    Gummy       Sun           Baylor         -0.0068493       0.68678             6\r\n    Gummy       Sun           Baylor             1.5326       0.18351            10\r\n    Gummy       Sloth         Chicago          -0.76967       0.36848             7\r\n    Gummy       Sloth         Chicago           0.37138       0.62562             9\r\n    Gummy       Spectacled    Coast Guard      -0.22558       0.78023             5\r\n    Gummy       Spectacled    Coast Guard        1.1174      0.081126             5\r\n    Gummy       Spectacled    Coast Guard       -1.0891       0.92939             9\r\n    Gummy       Sloth         Chicago          0.032557       0.77571             1\r\n    Gummy       Sloth         Chicago           0.55253       0.48679             2\r\n    Gummy       Spectacled    Coast Guard        1.1006       0.43586             2\r\n    Gummy       Spectacled    Coast Guard        1.5442       0.44678             4\r\n    Gummy       Spectacled    Coast Guard      0.085931       0.30635             9\r\n    Gummy       Spectacled    Coast Guard       -1.4916       0.50851             9\r\n    Gummy       Spectacled    Coast Guard       -0.7423       0.51077             1\r\n    Gummy       Spectacled    Coast Guard       -1.0616       0.81763             4\r\n    Gummy       Spectacled    Coast Guard        2.3505       0.79483             6\r\n    Gummy       Sloth         Chicago           -0.6156       0.64432             5\r\n    Gummy       Spectacled    Coast Guard       0.74808       0.37861             7\r\n    Gummy       Sloth         Chicago          -0.19242       0.81158             7\r\n    Gummy       Spectacled    Coast Guard       0.88861       0.53283             3\r\n    Gummy       Spectacled    Coast Guard      -0.76485       0.35073             5\r\n    Gummy       Spectacled    Coast Guard       -1.4023         0.939             1\r\n    Gummy       Black         Brown             -1.4224       0.87594            10\r\n    Gummy       Sloth         Chicago           0.48819       0.55016             2\r\n    Gummy       Sloth         Chicago          -0.17738       0.62248             2\r\n    Gummy       Sloth         Chicago          -0.19605       0.58704             4\r\n    Gummy       Sloth         Chicago            1.4193       0.20774             2\r\n    Gummy       Sloth         Chicago           0.29158       0.30125             5\r\n    Gummy       Sloth         Chicago           0.19781       0.47092             4\r\n    Gummy       Polar         Brown              1.5877       0.23049            10\r\n    Gummy       Polar         Brown            -0.80447       0.84431            10\r\n    Gummy       Sloth         Chicago           0.69662       0.19476             1\r\n    Gummy       Black         Brown             0.83509       0.22592             8\r\n    Gummy       Black         Brown            -0.24372       0.17071             3\r\n    Gummy       Sloth         Chicago           0.21567       0.22766             5\r\n    Gummy       Black         Brown             -1.1658        0.4357             6\r\n    Gummy       Sloth         Chicago            -1.148        0.3111            10\r\n    Gummy       Sloth         Chicago           0.10487       0.92338             5\r\n    Gummy       Sloth         Chicago           0.72225       0.43021            10\r\n    Gummy       Sloth         Chicago            2.5855       0.18482             4\r\n    Gummy       Sloth         Chicago          -0.66689       0.90488             8\r\n    Gummy       Sloth         Chicago           0.18733       0.97975             7\r\n    Gummy       Sloth         Chicago         -0.082494       0.43887             6\r\n    Gummy       Sloth         Chicago            -1.933       0.11112             7\r\n    Gummy       Sloth         Chicago          -0.43897       0.25806             7\r\n    Gummy       Sloth         Chicago           -1.7947       0.40872             2\r\n    Gummy       Sloth         Chicago           0.84038        0.5949             2\r\n    Gummy       Sloth         Chicago          -0.88803       0.26221            10\r\n    Gummy       Sloth         Chicago           0.10009       0.60284             2\r\n    Gummy       Sloth         Chicago          -0.54453       0.71122             1\r\n    Gummy       Sloth         Chicago           0.30352       0.22175             6\r\n    Gummy       Sloth         Chicago          -0.60033       0.11742             9\r\n    Gummy       Sloth         Chicago           0.48997       0.29668             7\r\n    Gummy       Sloth         Chicago           0.73936       0.31878             2\r\n    Gummy       Sloth         Chicago            1.7119       0.42417             4\r\n    Gummy       Sloth         Chicago          -0.19412       0.50786             5\r\n    Gummy       Sloth         Chicago           -2.1384      0.085516            10\r\n    Gummy       Sloth         Chicago          -0.83959       0.26248             2\r\n    Gummy       Sloth         Chicago            1.3546       0.80101             9\r\n    Gummy       Sloth         Chicago           -1.0722       0.02922             7\r\n    Gummy       Sloth         Chicago           0.96095       0.92885             4\r\n    Gummy       Sloth         Chicago           0.12405       0.73033             2\r\n    Gummy       Sloth         Chicago            1.4367       0.48861             5\r\n    Gummy       Sloth         Chicago           -1.9609       0.57853             5\r\n    Gummy       Black         Brown             -0.1977       0.23728             2\r\n    Gummy       Sloth         Chicago           -1.2078       0.45885             6\r\n    Gummy       Sloth         Chicago             2.908       0.96309             3\r\n    Gummy       Sloth         Chicago           0.82522       0.54681             4\r\n    Gummy       Sloth         Chicago             1.379       0.52114             6\r\n    Gummy       Sloth         Chicago           -1.0582       0.23159             3\r\n    Gummy       Sloth         Chicago          -0.46862        0.4889             3\r\n    Gummy       Black         Brown            -0.27247       0.62406             7\r\n    Gummy       Polar         Brown              1.0984       0.67914             3\r\n    Gummy       Sloth         Chicago          -0.27787       0.39552             9\r\n    Gummy       Sloth         Chicago           0.70154       0.36744            10\r\n    Gummy       Sloth         Chicago           -2.0518       0.98798             8\r\n    Gummy       Black         Brown            -0.35385      0.037739             4\r\n    Gummy       Sloth         Chicago          -0.82359       0.88517             6\r\n    Gummy       Sloth         Chicago           -1.5771       0.91329             2\r\n    Gummy       Sloth         Chicago           0.50797       0.79618            10\r\n    Gummy       Sloth         Chicago           0.28198      0.098712             9\r\n    Gummy       Sloth         Chicago           0.03348       0.26187             9\r\n    Gummy       Sloth         Chicago           -1.3337       0.33536             3\r\n    Gummy       Sloth         Chicago            1.1275       0.67973             6\r\n    Gummy       Sloth         Chicago           0.35018       0.13655             1\r\n    Gummy       Sloth         Chicago          -0.29907       0.72123             5\r\n    Gummy       Sloth         Chicago           0.02289       0.10676             4\r\n    Gummy       Sloth         Chicago            -0.262       0.65376             2\r\n    Gummy       Sloth         Chicago           -1.7502       0.49417             2\r\n    Gummy       Sloth         Chicago          -0.28565       0.77905             5\r\n    Gummy       Black         Brown            -0.83137       0.71504             1\r\n    Gummy       Sloth         Chicago          -0.97921       0.90372             6\r\n    Gummy       Sloth         Chicago           -1.1564       0.89092             5\r\n    Gummy       Sloth         Chicago          -0.53356       0.33416             7\r\n    Gummy       Sloth         Chicago           -2.0026       0.69875             7\r\n    Gummy       Sloth         Chicago           0.96423       0.19781             7\r\n    Gummy       Sloth         Chicago           0.52006      0.030541             1\r\n    Gummy       Sloth         Chicago         -0.020028       0.74407             1\r\n    Gummy       Sloth         Chicago         -0.034771       0.50002             4\r\n    Gummy       Sloth         Chicago          -0.79816       0.47992             6\r\n    Gummy       Sloth         Chicago            1.0187       0.90472             7\r\n    Gummy       Sloth         Chicago          -0.13322       0.60987             5\r\n    Gummy       Sloth         Chicago          -0.71453       0.61767             9\r\n    Gummy       Sloth         Chicago            1.3514       0.85944             8\r\n    Gummy       Sloth         Chicago          -0.22477       0.80549            10\r\n    Cinnamon    Polar         Brown            -0.58903       0.57672             6\r\n<\/pre>
The operation he was trying to calculate was the nan-omitted mean of Bytes based on two of the categories.<\/p>
[animalcandy, animal, candy] = findgroups(Bears.Animal,Bears.Candy);\r\nmeanbyte = splitapply(@(x)mean(x, 'omitnan'<\/span>), Bears.Bytes, animalcandy);\r\n<\/pre>
Error using vertcat\r\nDimensions of arrays being concatenated are not consistent.\r\nError in splitapply>localapply (line 257)\r\n            finalOut{curVar} = vertcat(funOut{:,curVar}); \r\nError in splitapply (line 132)\r\nvarargout = localapply(fun,splitData,gdim,nargout);\r\nError in mainDebuggingGroupedOps (line 32)\r\nmeanbyte = splitapply(@(x)mean(x, 'omitnan'), Bears.Bytes, animalcandy);<\/font><\/pre>
Hmm, I've seen that error before, but what does it have to do with this? How do we debug this?  One could put a break point at the anonymous function @(x)mean(x, 'omitnan')<\/tt> and then step with the debugger until the error occurs.<\/p>
\"\" <\/p>
This would likely work for a small number of groups, but as the number of groups gets larger, it would be lots of steps, one for each function evaluation.  You'd also likely have to do it twice, a second time after the error occurs.  Setting the debugger to stop on errors may work as well for splitapply<\/tt> but not for accumarray<\/tt> which is builtin and even with splitapply<\/tt> may not stop you in a useful spot.<\/p>
A trick I like to use is to just replace the function handle with {}. This takes whatever is provided and packs it into a cell so you can see exactly what is being passed into each function evaluation for each group.<\/p>
bytecell = splitapply(@(x){x}, Bears.Bytes, animalcandy);\r\ndisp(bytecell)\r\n<\/pre>
    [14×3 double]\r\n    [ 1×3 double]\r\n    [ 5×3 double]\r\n    [ 2×3 double]\r\n    [78×3 double]\r\n    [ 6×3 double]\r\n    [ 8×3 double]\r\n    [ 3×3 double]\r\n    [ 3×3 double]\r\n    [ 7×3 double]\r\n<\/pre>
From here we can see that the second cell has only one row.  Since mean takes the mean of the first non-singleton dimension, it's reducing this to a scalar by taking the mean of the row where the rest of the elements are coming rows from taking the mean of columns.  A scalar can't concatenate with a matrix so we get the error.<\/p>
The fix for this is simple, pass in the dimension to mean to force it to always take column mean.  Then rebuild the table with the labels.<\/p>
meanbyte = splitapply(@(x)mean(x, 1, 'omitnan'<\/span>), Bears.Bytes, animalcandy);\r\n\r\ndisp(table(animal, candy, meanbyte))\r\n<\/pre>
      animal       candy                   meanbyte              \r\n    __________    ________    ___________________________________\r\n    Spectacled    Gummy        0.075572      0.55805            5\r\n    Spectacled    Cinnamon       1.1093      0.45092            9\r\n    Sun           Gummy         -0.1449      0.42005          6.6\r\n    Sun           Cinnamon      0.03888      0.58718          5.5\r\n    Sloth         Gummy       -0.081113      0.51523       5.2949\r\n    Sloth         Cinnamon     -0.28948      0.55249       5.8333\r\n    Black         Gummy        -0.45653       0.4153        5.125\r\n    Black         Cinnamon     -0.33481      0.40734       1.3333\r\n    Polar         Gummy         0.62722      0.58464       7.6667\r\n    Polar         Cinnamon     -0.13228      0.32043       4.7143\r\n<\/pre>
In this case, the fix was fairly discernible from a quick inspection.  If it was not, we could loop over the cell and evaluate the function on each element to see where the error occurs.  If the error occurs on a specific cells' data, the loop will stop there and we can investigate.  If it's on the concatenate step, that'll be obvious at the end.<\/p>
fun = @(x)mean(x, 'omitnan'<\/span>);\r\nmeanbytecell = cell(size(bytecell));\r\nfor<\/span> ii = 1:numel(bytecell)\r\n    meanbytecell{ii} = fun(bytecell{ii});\r\nend<\/span>\r\ndisp(meanbytecell)\r\n<\/pre>
    [1×3 double]\r\n    [    3.5201]\r\n    [1×3 double]\r\n    [1×3 double]\r\n    [1×3 double]\r\n    [1×3 double]\r\n    [1×3 double]\r\n    [1×3 double]\r\n    [1×3 double]\r\n    [1×3 double]\r\n<\/pre>
And now it is obvious why these won't concatenate.<\/p>
I find looping over the cell in this manner to be much easier than looping over the original data set and trying to identify which elements are in which groups and indexing correctly.<\/p>
I'm also a big fan of using splitapply\/accumarray<\/tt> with cell output for making objects or plots based on grouped data where the object can't be returned directly.  Continuing this example we'll use a histogram for each group of original data, wrapping histogram<\/tt> in {}.<\/p>
figure\r\naxes('ColorOrder'<\/span>, parula(numel(animal)))\r\nhold on<\/span>\r\nh = splitapply(@(x){histogram(x)}, Bears.Bytes, animalcandy);\r\nlegend([h{:}], compose(\"%s\/%s\"<\/span>, animal, candy));\r\n<\/pre>\"\" 
On an aside, development has been working to make grouped operations easier over the last few releases with a collection of new functions:<\/p>