Writing a Script to Count Data in Multiple Text Files

Posted by Stuart McGarrity, November 8, 2017

3 views (last 30 days) | 0 Likes | 2 comments

Here I need to combine the data from multiple text files by grouping and summing by one variable. I also try using a datastore which I have not used much before.

Features covered in this code-along style video include:

readtable, fileDatastore with read function parameter
varfun

Video Player is loading.

Current Time 0:00

Duration 37:54

Loaded: 0.44%

Stream Type LIVE

Remaining Time 37:54

Play the video in full screen mode for a better viewing experience.

%% Read Redirect Traffic from Server Logs

%% Parameters
folderName='C:\Folder';
outputFileName='Redirects4Weeks.csv';

%% Get List of Files
fileInfo=dir([folderName filesep '*.csv']);
fileName={fileInfo.name}';
fullFileNames=cellfun(@(x) [folderName filesep x],fileName,'UniformOutput', false);
fileDatesCell=regexp(fileName,'(\d+-\d+-\d+)','tokens');
fileDatesStr=stripCell(fileDatesCell,2);
fileDates=datetime(fileDatesStr);
[fileDates,idx]=sort(fileDates,'descend');
mostRecent4=idx(1:4);

%% Data Store
fcn=@(x) readtable(x,'ReadVariableNames',false,'Delimiter',',','TextType','string', 'HeaderLines',1);
ds = fileDatastore(fullFileNames(mostRecent4),'ReadFcn', fcn);
cellData=readall(ds);
data=vertcat(cellData{:});
data.Properties.VariableNames={'Request_URI' 'count'};

%% Count URLs
pivot=varfun(@sum,data,'InputVariables','count','GroupingVariables',{'Request_URI'});
pivot.GroupCount=[];
pivot.Properties.VariableNames{'sum_count'}='count';
pivot=sortrows(pivot,{'count'},{'descend'});

%% Save
writetable(pivot,outputFileName);