{"id":912,"date":"2013-11-03T19:20:25","date_gmt":"2013-11-04T00:20:25","guid":{"rendered":"https:\/\/blogs.mathworks.com\/steve\/?p=912"},"modified":"2019-11-01T09:34:08","modified_gmt":"2019-11-01T13:34:08","slug":"chess-and-a-little-text-file-manipulation","status":"publish","type":"post","link":"https:\/\/blogs.mathworks.com\/steve\/2013\/11\/03\/chess-and-a-little-text-file-manipulation\/","title":{"rendered":"Chess and a little text file manipulation"},"content":{"rendered":"\r\n

Here's an image of a chess position:<\/p>

\"\" <\/p>

And that's about as close to image processing as today's blog post will come. Because this post is really about text processing.<\/p>

It seems like a lot of computational tasks in engineering and science involve manipulating data in text files. This weekend I had such a task, although I must admit that it had nothing to do with engineering or science. Even so, I thought the task would be a good illustration of some basic text processing techniques.<\/p>

I have a text file, tactics.pgn, that is a database of chess tactics puzzles. Here are lines 2,759 through 2,784 from the file.<\/p>

[Event \"?\"]\r\n[Site \"?\"]\r\n[Date \"????.??.??\"]\r\n[Round \"?\"]\r\n[White \"Diagram 211\"]\r\n[Black \"\"]\r\n[Result \"*\"]\r\n[EventDate \"2006.05.20\"]\r\n[FEN \"2b1R3\/ppk4p\/8\/2q2p2\/2Br2n1\/2QP2N1\/P4PPP\/6K1 w - - 0 1\"]\r\n[SetUp \"1\"]\r\n[SourceDate \"2011.02.22\"]<\/pre>
1.Rxc8+ Kxc8 2.Be6+ Kd8 3.Qxc5 *<\/pre>
[Event \"?\"]\r\n[Site \"?\"]\r\n[Date \"????.??.??\"]\r\n[Round \"?\"]\r\n[White \"Diagram 212\"]\r\n[Black \"\"]\r\n[Result \"*\"]\r\n[FEN \"3rr1k1\/p1p2ppp\/Q2b1q2\/8\/3Np3\/4P3\/PP3PPP\/R1B2RK1 b - - 0 1\"]\r\n[SetUp \"1\"]\r\n[SourceDate \"2011.02.22\"]<\/pre>
1...Bxh2+ 2.Kxh2 Qxa6 *<\/pre>
These lines store two chess positions. The first is the position I showed above, with White to move. Can you figure out how White wins by capturing the Bishop on c8 with his rook? (If you're interested in how the position is encoded in the text, see the Wikipedia article on Forsyth-Edwards Notation<\/a>, or FEN.)<\/p>
I wanted to shuffle the positions in this file randomly. None of the chess software programs that I have can do this, so I decided to tackle it with MATLAB. (Fun thing to do on a Saturday morning, right?)<\/p>
OK, so here's the basic procedure:<\/p>
  1. Read the entire file into MATLAB.<\/li>
  2. Split the data into chunks, one chunk for each position.<\/li>
  3. Rearrange the chunks randomly.<\/li>
  4. Write out the rearranged chunks to a new text file.<\/li><\/ol><\/div>
    There are a lot of different ways to do this. Here's what I came up with.<\/p>
    First, read in the entire file. The function fileread<\/tt> is just the ticket.<\/p>
    characters = fileread('tactics.pgn'<\/span>);\r\nsize(characters)\r\n<\/pre>
    \r\nans =\r\n\r\n           1      106394\r\n\r\n<\/pre>
    You can see that there are about 106,000 characters in the file. Let's split the data into lines using strsplit<\/tt>.<\/p>
    lines = strsplit(characters,'\\n'<\/span>)';\r\nsize(lines)\r\n<\/pre>
    \r\nans =\r\n\r\n        4838           1\r\n\r\n<\/pre>
    There are about 4,800 lines of text. But how many positions are there? I'm going to find the starting line of each position by searching for the string \"[Event \" at the beginning a line. It's time for regexp<\/tt>.<\/p>
    idx = regexp(lines,'^[Event '<\/span>);\r\nidx(1:15)\r\n<\/pre>
    \r\nans = \r\n\r\n    [1]\r\n    []\r\n    []\r\n    []\r\n    []\r\n    []\r\n    []\r\n    []\r\n    []\r\n    []\r\n    []\r\n    [1]\r\n    []\r\n    []\r\n    []\r\n\r\n<\/pre>
    This shows us that this string is found twice in the first 15 lines of the file. idx<\/tt> is a cell array, so I'll use cellfun<\/tt> and find<\/tt> to identify all the lines that contain the matching string. Each of these lines is the start of an entry for one chess position.<\/p>
    first_lines = find(~cellfun(@isempty,idx));\r\nfirst_lines(1:3)\r\n<\/pre>
    \r\nans =\r\n\r\n     1\r\n    12\r\n    23\r\n\r\n<\/pre>
    So there are positions starting on lines 1, 12, and 23.<\/p>
    Next, I'll make a cell array such that that each cell contains all the lines for one position. To make the for-loop work, I'm going to add an \"extra\" value to the first_lines vector that points to a nonexistent line just past the of the file.<\/p>
    first_lines(end+1) = length(lines) + 1;\r\nfor<\/span> k = 1:length(first_lines)-1\r\n    positions{k} = lines(first_lines(k):first_lines(k+1)-1);\r\nend<\/span>\r\n<\/pre>
    Let's take a look at what we have now.<\/p>
    size(positions)\r\n<\/pre>
    \r\nans =\r\n\r\n     1   421\r\n\r\n<\/pre>
    There are 421 positions in the file. For example:<\/p>
    positions{205}\r\n<\/pre>
    \r\nans = \r\n\r\n    '[Event \"?\"]'\r\n    '[Site \"?\"]'\r\n    '[Date \"????.??.??\"]'\r\n    '[Round \"?\"]'\r\n    '[White \"Diagram 211\"]'\r\n    '[Black \"Bain\"]'\r\n    '[Result \"*\"]'\r\n    '[EventDate \"2006.05.20\"]'\r\n    '[FEN \"2b1R3\/ppk4p\/8\/2q2p2\/2Br2n1\/2QP2N1\/P4PPP\/6K1 w - - 0 1\"]'\r\n    '[SetUp \"1\"]'\r\n    '[SourceDate \"2011.02.22\"]'\r\n    '1.Rxc8+ Kxc8 2.Be6+ Kd8 3.Qxc5 *'\r\n\r\n<\/pre>
    Again, this is the position shown at top of this post.<\/p>
    Getting near the end, now. It's time to rearrange the positions. Before I do that, though, I'll shuffle the random number generator. I only do this so that if I repeat these steps in a new MATLAB session, I'll be sure to get a different result. After shuffling the random number generator using rng<\/tt>, a quick call to randperm<\/tt> randomly rearranges the positions.<\/p>
    rng shuffle<\/span>\r\nshuffled_positions = positions(randperm(length(positions)));\r\n<\/pre>
    We've arrived at the last step: writing out the shuffled positions to a new file.<\/p>
    fid = fopen('shuffled_tactics.pgn'<\/span>,'w'<\/span>);\r\nfor<\/span> k = 1:length(shuffled_positions)\r\n    position = shuffled_positions{k};\r\n    for<\/span> p = 1:length(position)\r\n        fprintf(fid,'%s\\n'<\/span>,position{p});\r\n    end<\/span>\r\nend<\/span>\r\nfclose(fid);\r\n<\/pre>
    And that's it!<\/p>
    Reading text, manipulating it in some useful way, and writing the results back out -- a common computing task accomplished using several basic MATLAB functions.<\/p>