Hannes Wins!

May 6th, 2010

In the final minutes all the secret tricks, squirreled away for the last desperate sprint, come flying into the daylight at last. We saw strong finishes from Sergey Y., Yi Cao, Abhisek Ukil, and srach. But in the end, Hannes Naudé prevailed, closing the contest with a smashing blow that knocked a whopping tenth of a percent off the score. In the past Hannes has won (together with his sometimes comrade-in-arms Cobus Potgieter) two mid-contest prizes, but this is his first Grand Prize. His hauntingly named Overfitting is a fool’s game stayed atop the grinding queue for hours, daring anyone to displace it. None could.

Congratulations, Hannes, on a job well done!

On the newsgroup, Hannes says some of his techniques included “intentionally overfitting the dataset, something for which I have a deep seated dislike”. I hope we can get him to write about his winning entry along with a digression on the nature of robustness and brittleness in machine learning. It’s clear that the contest is maddeningly prone to overfitting, but it’s also hard to banish overfitting from any contest or benchmark. As always, we are looking for ways to make the next contest better, and now is a good time to reflect on things we could improve about the interface and the problem. How might we encourage a richer exploration of the solution space without damaging the fun back-and-forth nature of tweak battles? Another big puzzle for us here is how to make the contest more appealing to the confused and curious crowds who watch from the sidelines. How would you turn the contest into a better spectator sport? Tell us in the newsgroup, or here in the comments, or use the User Voice page we set up for this purpose.

As always, thanks to everyone for joining us, and we hope to see you next time.

By Ned Gulley

Ned is part of the MATLAB Central team.

You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.

7 Responses to “Hannes Wins!”

Alan Chalker replied on May 6th, 2010 at 04:42 UTC :

Thanks again for an exciting and fun filled week! I’d like to provide some suggestions:

1. Regarding ‘spectator’ involvement, one option is to utilize the User Voice page to allow everyone to vote on something, which could be another ‘mini-contest’. For example, maybe the ‘power competitors’ could be encouraged to submit a particular entry to the User Voice page that is well documented or provides a elegant solution in a very small number of lines or perhaps visualizes step by step what a solver is doing. This would then serve as a way that ‘novice competitors’ could easily start to understand what’s going on.

2. Another suggestion would be to solicit from the user community special test boards to be added to the test suite during the late stages of the contest. Many people might not understand the actual algorithms, but might be able to come up with challenging test entries. For example, in the current contest, I think it would have been interesting to see how the solvers handle artificially created patterns like zig zags or concentric rings.

3. One way to maybe address both the ‘over-fitting’ and the novice user is to have entries initially run against only a subset of the test suite. For example, say there are 100 images in the suite. Make my entries only run against images 1:50, Oliver’s against 2:51, srach’s against 3:51, etc. etc. This would force some cooperation between people and would allow ‘novice players’ to perhaps bring something to the table the rest of us don’t have (i.e. access to images in the test suite). After the queue closed, you could then run the leading entries against the entire test suite to see how well they generalized.

4. Provide some sort of token award to anyone who submits an entry that meets some criteria (perhaps it finishes in the top 50% of the current entries?). Maybe it’s just a special badge on their profile or a Matlab decal you mail to them, but whatever it is it could incentivize the community.

5. Do some ‘viral marketing’ of the contest when it starts to drum up more interest. I first discovered it years ago when someone posted a comment about it on Slashdot. Maybe try submitting stories to the big tech sites like Slashdot, Digg, BoingBoing, Engadget, etc. etc.

6. Allow somehow for the designation of teams of up to a certain size and then award a prize based upon best overall team performance, such as average percent improvement or most time taking the lead.

7. Award a prize for the person with the best average rank at the end of the event. This would severely penalize someone like myself who submits hundreds of entries, but would appeal to those who thoughtfully analyze the algorithms and submit only a few entries with improvements.

8. Provide a new interface / descriptions to the statistics page (and make sure it updates properly;). This could end up being the main entry point into the contest area, where spectators could get a ‘snapshot’ of what’s going on with the contest and then dive into to specific entries, users, or time blocks depending on their level of interest.

9. Provide a toolbox that integrates the contest website into the MATLAB desktop, much like the new file exchange browser is integrated. This would allow people to do most everything entirely in that window, instead of having to switch to a browser constantly.

10. Tell that Alan Chalker guy to stop being so wordy and let other people chime in now and then;)

Hannes Naudé replied on May 6th, 2010 at 07:09 UTC :

Little suggestion/request from my side. Would it maybe be possible to provide the contest dataset for us to analyze. I don’t mean the testsuite, I mean the database of entries, their scores etc like the Peg Solitaire database that was distributed for the DataVis contest. I had some interesting thoughts on constructing diagrams allowing us to view entries as points in a 2D scatterplot with similar entries clustering together etc. I’d like to pursue this for a bit, and if I produce anything of value I will share it (it might even make a worthwhile addition to the stats page in future).

Come to think of it, if you have this data in .mat format anyway (I assume all contest machinery is written in Matlab ;-)) then it should be possible to share it for direct download even while a contest is live. This will open up enticing new posibilities for monitoring the contest for interesting but low scoring new entries, stealthy probes or opportunities for combining ideas from other competitors. Of course this is possible today, but requires web-scraping, which is something most of us shy away from.

EBS replied on May 6th, 2010 at 23:40 UTC :

Great contest team!

Ned, what about increasing the number of votes allowed on the UserVoice site? I ran out of entries and there are still useful ideas to vote on – even though I only gave one vote per good entry! This also means that I can’t submit any ideas of my own :(

I think the # of votes could be easily doubled or tripled…

Cheers,
EBS

Ned replied on May 7th, 2010 at 16:23 UTC :

@EBS

Unfortunately the service we’re using (UserVoice) doesn’t permit this.

EBS replied on May 7th, 2010 at 16:39 UTC :

@Ned:

You can set the number of votes per user if you use their ‘Bronze’ or higher service plans :)

Igor Carron replied on May 12th, 2010 at 07:46 UTC :

Ned,

Do you think you could release the test suite ?

Igor.
http://nuit-blanche.blogspot.com