Difference between revisions of "Judge:History"

Revision as of 23:15, 6 December 2014

Background

Construction of the PEG Judge began during the 2007/2008 school year, as a project by Hanson and Guru, two former PEG leaders, to help Ms. Plachta by writing an autograder for the 11/12 enriched CS class at Woburn.

Ms. Plachta demanded the highest standards of achievement from her students, in order to push them to reach their full potential. A student could typically earn a 99 in the course by earning a large number of points (on the order of magnitude of 1000) in one school year, where a single CS problem was typically worth between 3 and 20 points (depending on difficulty). This translated to lots and lots of marking for Ms. Plachta, who had to download the source file, save it in the student's folder, compile the program, locate the test data, run the program once for each case, then compare its output with the expected output for each case. If the program failed, she often tried to figure out why, to give feedback to the student, and if it worked, she had to update the points spreadsheet. In the code submission frenzies that preceded report card mark submission deadlines, Ms. Plachta often had to work day and night to finish all her marking.

When Ms. Plachta became ill with cancer, it became clear that this system was unsustainable. There were students in the class experienced enough to give feedback to other students who were having trouble with getting their code to work, but it would have been impossible for them to mark everyone’s programs; it would have been too much work for these students who had their own work to do too. Meanwhile, computers are very bad at figuring out humans' mistakes, but very good at all the other tasks involved in the marking process (automatic submission, compilation, execution, and grading). And so the PEG Judge was born, and complemented the efforts of Guru, Hanson, Jacob, and others in taking over the teaching of the class.

Genesis

On September 23rd, 2008, the first source file was submitted to the PEG Judge, which at the time supported only C and C++ (with Pascal being added soon afterward). The Judge was hosted on a machine on loan from the school, a Dell OptiPlex with a dual-core Intel CPU and 4MB L2 cache, running 32-bit Ubuntu GNU/Linux. The internet connection was provided by Jacob, who connected the Judge to his home internet service with Rogers. Since hosting servers on residential internet contracts is typically not allowed, Hanson decided to run the server on some port other than 80 (the standard for the HTTP), in case Rogers was monitoring unauthorized activity on port 80. Port number 5050 was chosen for this purpose, as almost all other ports were filtered. Also, he found a simple dynamic DNS setup, and so the original URL of the Judge was http://pegjudge.ath.cx:5050/. The Judge at that time was very new; it had a very basic set of features, but it got the job done. Guru wrote most of the back end, and Hanson wrote most of the front end. The core grader, responsible for setting up a sandbox and measuring resource usage, was taken from Mooshak, as no PEG member at the time had the expertise in Unix-based systems necessary to write a grader from scratch.

Development

The Judge saw a dramatic improvement in the interface over the course of the 2008/2009 school year as Hanson completely overhauled the front end, leaving the interface cleaner, crisper, and more professional-looking. He also overhauled the back end, switching from the ad hoc PHP functions being used before to the secure and high-performance CodeIgniter framework. (The grader, however, was only tweaked slightly.)

By the end of that year the Judge in its current form was essentially complete, a good thing as Hanson was off to university and would likely have other things on his mind. It even included support for running contests and posting comments. Brian, who had been helping with administration since the beginning of the year, took over at this point as primary administrator of the Judge. Brian's primary contributions were the addition of this wiki to the site and support for various additional popular programming languages.

Later years

With the passing of Ms. Plachta, the Judge was never as popular as before, for students were no longer being pushed to earn 1000-odd points per year. But there were still some PEG members using it; Canada’s IOI teams trained on it; and Brian and a friend from Waterloo C.I. ran a mock DWITE contest. The decision was thus made to keep the Judge alive. When the Plachta house was sold in June 2010, Jacob could no longer host the Judge, so the Judge was migrated to Brian's laptop; this hosting was again illegitimate. When Brian left for university at the beginning of September 2010, he could not risk violating the residence contract that stated that hosting servers was strictly disallowed, so the Judge was left without hosting for a few weeks. Brian, Jacob, and Hanson were hopeful about convincing the Center for Education in Mathematics and Computing at the University of Waterloo to grant PEG hosting, for this is the organization primarily involved with high achievement in computer science at the high school level, running the Canadian Computing Competition and sending Team Canada to the IOI. Almost all problems from CCCs of years past, both stage 1 and stage 2, were already in the PEG Judge's archive, the interface had taken two years of work, the grader had been tweaked precisely, and, all in all, the PEG Judge was the first in Canada with a level of sophistication on par with those of the major online judges of the time. This did not work out, so Brian started looking for hosting companies.

Recent hosting

The Judge was migrated a second time at the end of September 2010 to a OpenVZ box hosted by IntoVPS, which offered 512 MB guaranteed RAM, 30 GB disk space, 500 GB monthly bandwidth, and a quad-core CPU for only $10 USD/month. The :5050 suffix on the Judge URL was dropped, and the web server simply bound to port 80, since the hosting was then legitimate; Jacob also acquired the wcipeg.com domain from James Rosselet (a former PEG member), giving the URL of the Judge as the current http://wcipeg.com/.

A month later, it was again migrated to hosting by Virpus, which offered 1 GB guaranteed RAM, 50 GB disk space, 1600 GB monthly bandwidth, and six CPU cores, as Brian thought this was a better plan.

However, Virpus hosting suffered from too much downtime, so in June of 2011 the Judge was migrated a fourth time to a VolumeDrive box.

In October of 2011, the Judge was migrated a fifth time to hosting offered by LoveVPS, this time a Xen box, due to Brian's dissatisfaction with VolumeDrive's support and the incomplete virtualization of OpenVZ containers in general.

Although LoveVPS was a satisfactory VPS host, Brian was again forced to migrate the Judge in May 2013, this time to HostingBeam (then PhoenixVPS); the reasons for this move will not be discussed here. HostingBeam suffered from strange performance issues: the virtualized file system was very slow, and so was ptrace, an important operating system API used for automatic judging. These issues could be temporarily fixed by rebooting, but it would have been necessary to reboot the VPS more than once a week in order to maintain satisfactory performance.

Brian ultimately could not diagnose these issues with the HostingBeam VPS, so around June 2014, the Judge was moved to a Linode plan with 2 GB of RAM. (It is worth noting that Linode is generally considered one of the best VPS hosts, but was much more expensive around 2010; it became more feasible to host an online judge on Linode after April 2014, when the company doubled the RAM on all of their plans.)

Future plans

Brian plans to rewrite the core grader from scratch, to improve support for languages such as Java that invariably use multiple processes (something that the Unix API does not handle well). He also plans to refactor and then open-source the Judge code. It's currently uncertain whether either of these projects will ever see the light of day.