Difference between revisions of "Judge:History"

From PEGWiki
Jump to: navigation, search
(Created page with '==A brief history of the Judge== During the 2007/2008 school year, Hanson and Guru decided to devote some time to a most worthy cause: writing an autograder for the 11/12 enriche…')
 
(→‎Future plans)
 
(3 intermediate revisions by the same user not shown)
Line 1: Line 1:
−
==A brief history of the Judge==
+
==Background==
−
During the 2007/2008 school year, Hanson and Guru decided to devote some time to a most worthy cause: writing an autograder for the 11/12 enriched CS class at Woburn. In order to encourage us to develop her students' problem-solving skills as much as possible, Ms. Plachta set high expectations for them, such as earning 1000 points by the end of the year, where a typical problem is worth between 3 and 20 points. Students who wanted to earn a 99 in the course then had to solve between 50 and 300 problems, which translated to lots and lots of marking for Ms. Plachta. Marking programs is very tedious work. She had to download the program, save it in the student's folder, compile the program, locate the test data, run the program once for each case, then compare its output with the expected output. If the program failed, she often tried to figure out why, to give feedback to the student, and if it worked, she had to update the points spreadsheet.
+
Construction of the PEG Judge began during the 2007/2008 school year, as a project by Hanson and Guru, two former [[PEG:PEG leaders|PEG leaders]], to help Ms. Plachta by writing an autograder for the 11/12 enriched CS class at Woburn.
  
−
The last task is the only one that should require human intervention, and it was clear that Ms. Plachta was spending far too much time on the other tasks. In the periods of time immediately preceding report card mark submission deadlines she was literally spending day and night marking, marking, marking. As she was in poor health due to the cancer it became clear that this would be too much of a burden for her. The more experienced students in class could give feedback to students whose programs didn't work, but it would have been impossible for them to mark everyone’s programs; it would have been too much work for students who had their own work to do too. And so the PEG Judge was born. Though its creators did not anticipate this, the Judge would have the added benefit of encouraging students to code.
+
Ms. Plachta demanded the highest standards of achievement from her students, in order to push them to reach their full potential. A student could typically earn a 99 in the course by earning a large number of points (on the order of magnitude of 1000) in one school year, where a single CS problem was typically worth between 3 and 20 points (depending on difficulty). This translated to lots and lots of marking for Ms. Plachta, who had to download the source file, save it in the student's folder, compile the program, locate the test data, run the program once for each case, then compare its output with the expected output for each case. If the program failed, she often tried to figure out why, to give feedback to the student, and if it worked, she had to update the points spreadsheet. In the code submission frenzies that preceded report card mark submission deadlines, Ms. Plachta often had to work day and night to finish all her marking.
  
−
On September 23rd, 2008, the first source file was submitted to the PEG Judge, which at the time supported only C and C++ (with Pascal being added soon afterward). The Judge was hosted on one of the school’s machines, a Dell OptiPlex with a dual-core Intel CPU and 4MB L2 cache using Jacob’s home internet connection with Rogers. As hosting servers on residential internet contracts is typically not allowed, Hanson decided to run the server on some port other than 80, in case the ISP was scanning for unauthorized activity on port 80. After trying various ports he determined that 5050 is the only port above the reserved range (''i.e.'', the range 1-1023 to which, on some operating systems, permission to bind is withheld by the kernel and granted only to users with sufficient privilege) that isn't actually filtered by the school board's firewall. So there we sat at http://pegjudge.ath.cx:5050/ . The Judge at that time was very new; it had a very basic set of features, but it got the job done. Guru wrote most of the back end, particularly the grader itself, and Hanson wrote most of the front end.
+
When Ms. Plachta became ill with cancer, it became clear that this system was unsustainable. There were students in the class experienced enough to give feedback to other students who were having trouble with getting their code to work, but it would have been impossible for them to mark everyone’s programs; it would have been too much work for these students who had their own work to do too. Meanwhile, computers are very bad at figuring out humans' mistakes, but very good at all the other tasks involved in the marking process (automatic submission, compilation, execution, and grading). And so the PEG Judge was born, and complemented the efforts of Guru, Hanson, Jacob, and others in taking over the teaching of the class.
  
−
The 2008/2009 school year saw a dramatic improvement in the interface as Hanson completely overhauled the front end, leaving the interface cleaner, crisper, and more professional-looking. He also overhauled the back end, switching from the ''ad hoc'' PHP functions being used before to the CodeIgniter framework. (The grader, though, has been tweaked only slightly since the inception of the Judge.)
+
==Genesis==
 +
On September 23rd, 2008, the first source file was submitted to the PEG Judge, which at the time supported only C and C++ (with Pascal being added soon afterward). The Judge was hosted on a machine on loan from the school, a Dell OptiPlex with a dual-core Intel CPU and 4MB L2 cache, running 32-bit Ubuntu GNU/Linux. The internet connection was provided by Jacob, who connected the Judge to his home internet service with Rogers. Since hosting servers on residential internet contracts is typically not allowed, Hanson decided to run the server on some port other than 80 (the standard for the HTTP), in case Rogers was monitoring unauthorized activity on port 80. Port number 5050 was chosen for this purpose, as almost all other ports were filtered. Also, he found a simple dynamic DNS setup, and so the original URL of the Judge was <code>http://pegjudge.ath.cx:5050/</code>. The Judge at that time was very new; it had a very basic set of features, but it got the job done. Guru wrote most of the back end, and Hanson wrote most of the front end. The core grader, responsible for setting up a sandbox and measuring resource usage, was taken from [http://mooshak.dcc.fc.up.pt/ Mooshak], as no PEG member at the time had the expertise in Unix-based systems necessary to write a grader from scratch.
  
−
By the end of that year the Judge in its current form was essentially complete, a good thing as Hanson was off to university and would likely have other things on his mind. As Brian had started using Linux the previous year, he was in an ideal position to take over the administration of the Judge (but he had to trick Hanson into giving him root access). In November he added a wiki to the site and he has also gradually populated the allowed languages list.
+
==Development==
 +
The Judge saw a dramatic improvement in the interface over the course of the 2008/2009 school year as Hanson completely overhauled the front end, leaving the interface cleaner, crisper, and more professional-looking. He also overhauled the back end, switching from the ''ad hoc'' PHP functions being used before to the secure and high-performance CodeIgniter framework. (The grader, however, was only tweaked slightly.)
  
−
With the passing of Ms. Plachta the Judge was never as popular as before, for students were no longer being pushed to earn 1000 points. But there were still some PEG members using it; Canada’s IOI teams trained on it; and Brian and a friend from Waterloo C.I. tried to run a mock DWITE contest. And so the Judge went on. When the Plachta house was sold in June, the Judge was migrated to Brian's laptop, again illicitly. When Brian left for university, he could not risk violating the residence contract, so the Judge was left without hosting. Brian, Jacob, and Hanson were hopeful about convincing the Center for Education in Mathematics and Computing at the University of Waterloo to grant PEG hosting, for this is the organization primarily involved with high achievement in computer science at the high school level, running the Canadian Computing Competition and sending Team Canada to the IOI. Almost all problems from CCCs of years past, both stage 1 and stage 2, were already in the PEG Judge's archive, the interface had taken two years of work, the grader had been tweaked precisely, and, all in all, the PEG Judge was the first in Canada with a level of sophistication on par with those of the major online judges of the time. PEG's request was denied by the CEMC, forcing Brian to look elsewhere.
+
By the end of that year the Judge in its current form was essentially complete, a good thing as Hanson was off to university and would likely have other things on his mind. It even included support for running contests and posting comments. Brian, who had been helping with administration since the beginning of the year, took over at this point as primary administrator of the Judge. Brian's primary contributions were the addition of this wiki to the site and support for various additional popular programming languages.
  
−
Luckily, some very good VPS plans exist; [http://www.intovps.com/ IntoVPS], for example, offers 512 MB guaranteed RAM, 30 GB disk space, 500 GB monthly bandwidth, and a quad-core CPU for only $10 USD/month. The PEG Judge was initially migrated to an IntoVPS server; a month later it was again migrated to hosting by [http://www.virpus.com/ Virpus], which offers 1 GB guaranteed RAM, 50 GB disk space, 1600 GB monthly bandwidth, and six CPU cores. As hosting is now legitimate, the :5050 suffix was dropped and the site is now http://pegjudge.ath.cx/
+
==Later years==
 +
With the passing of Ms. Plachta, the Judge was never as popular as before, for students were no longer being pushed to earn 1000-odd points per year. But there were still some PEG members using it; Canada’s IOI teams trained on it; and Brian and a friend from Waterloo C.I. ran a mock DWITE contest. The decision was thus made to keep the Judge alive. When the Plachta house was sold in June 2010, Jacob could no longer host the Judge, so the Judge was migrated to Brian's laptop; this hosting was again illegitimate. When Brian left for university at the beginning of September 2010, he could not risk violating the residence contract that stated that hosting servers was strictly disallowed, so the Judge was left without hosting for a few weeks. Brian, Jacob, and Hanson were hopeful about convincing the Center for Education in Mathematics and Computing at the University of Waterloo to grant PEG hosting, for this is the organization primarily involved with high achievement in computer science at the high school level, running the Canadian Computing Competition and sending Team Canada to the IOI. Almost all problems from CCCs of years past, both stage 1 and stage 2, were already in the PEG Judge's archive, the interface had taken two years of work, the grader had been tweaked precisely, and, all in all, the PEG Judge was the first in Canada with a level of sophistication on par with those of the major online judges of the time. This did not work out, so Brian started looking for hosting companies.
 +
 
 +
==Recent hosting==
 +
The Judge was migrated a second time at the end of September 2010 to a OpenVZ box hosted by [http://www.intovps.com/ IntoVPS], which offered 512 MB guaranteed RAM, 30 GB disk space, 500 GB monthly bandwidth, and a quad-core CPU for only $10 USD/month. The :5050 suffix on the Judge URL was dropped, and the web server simply bound to port 80, since the hosting was then legitimate; Jacob also acquired the <code>wcipeg.com</code> domain from James Rosselet (a former PEG member), giving the URL of the Judge as the current {{Root}}.
 +
 
 +
A month later, it was again migrated to hosting by [http://www.virpus.com/ Virpus], which offered 1 GB guaranteed RAM, 50 GB disk space, 1600 GB monthly bandwidth, and six CPU cores, as Brian thought this was a better plan.
 +
 
 +
However, Virpus hosting suffered from too much downtime, so in June of 2011 the Judge was migrated a fourth time to a [http://volumedrive.com/ VolumeDrive] box.
 +
 
 +
In October of 2011, the Judge was migrated a fifth time to hosting offered by [http://lovevps.com/ LoveVPS], this time a Xen box, due to Brian's dissatisfaction with VolumeDrive's support and the incomplete virtualization of OpenVZ containers in general.
 +
 
 +
Although LoveVPS was a satisfactory VPS host, Brian was again forced to migrate the Judge in May 2013, this time to HostingBeam (then PhoenixVPS); the reasons for this move will not be discussed here. HostingBeam suffered from strange performance issues: the virtualized file system was very slow, and so was <code>ptrace</code>, an important operating system API used for automatic judging. These issues could be temporarily fixed by rebooting, but it would have been necessary to reboot the VPS more than once a week in order to maintain satisfactory performance.
 +
 
 +
Brian ultimately could not diagnose these issues with the HostingBeam VPS, so around June 2014, the Judge was moved to a [http://www.linode.com/ Linode] plan with 2 GB of RAM. (It is worth noting that Linode is generally considered one of the best VPS hosts, but was much more expensive around 2010; it became more feasible to host an online judge on Linode after April 2014, when the company doubled the RAM on all of their plans.)
 +
 
 +
==Future plans==
 +
Brian plans to rewrite the core grader from scratch, to improve support for languages such as Java that invariably use multiple processes (something that the Unix API does not handle well). He also plans to refactor and then open-source the Judge code. It's currently uncertain whether either of these projects will ever see the light of day.
 +
 
 +
Brian also hopes that Google Fiber will one day come to Silicon Valley, where he currently resides, so that he can host the Judge on a dedicated server in his own home.

Latest revision as of 23:16, 6 December 2014

Background

Construction of the PEG Judge began during the 2007/2008 school year, as a project by Hanson and Guru, two former PEG leaders, to help Ms. Plachta by writing an autograder for the 11/12 enriched CS class at Woburn.

Ms. Plachta demanded the highest standards of achievement from her students, in order to push them to reach their full potential. A student could typically earn a 99 in the course by earning a large number of points (on the order of magnitude of 1000) in one school year, where a single CS problem was typically worth between 3 and 20 points (depending on difficulty). This translated to lots and lots of marking for Ms. Plachta, who had to download the source file, save it in the student's folder, compile the program, locate the test data, run the program once for each case, then compare its output with the expected output for each case. If the program failed, she often tried to figure out why, to give feedback to the student, and if it worked, she had to update the points spreadsheet. In the code submission frenzies that preceded report card mark submission deadlines, Ms. Plachta often had to work day and night to finish all her marking.

When Ms. Plachta became ill with cancer, it became clear that this system was unsustainable. There were students in the class experienced enough to give feedback to other students who were having trouble with getting their code to work, but it would have been impossible for them to mark everyone’s programs; it would have been too much work for these students who had their own work to do too. Meanwhile, computers are very bad at figuring out humans' mistakes, but very good at all the other tasks involved in the marking process (automatic submission, compilation, execution, and grading). And so the PEG Judge was born, and complemented the efforts of Guru, Hanson, Jacob, and others in taking over the teaching of the class.

Genesis

On September 23rd, 2008, the first source file was submitted to the PEG Judge, which at the time supported only C and C++ (with Pascal being added soon afterward). The Judge was hosted on a machine on loan from the school, a Dell OptiPlex with a dual-core Intel CPU and 4MB L2 cache, running 32-bit Ubuntu GNU/Linux. The internet connection was provided by Jacob, who connected the Judge to his home internet service with Rogers. Since hosting servers on residential internet contracts is typically not allowed, Hanson decided to run the server on some port other than 80 (the standard for the HTTP), in case Rogers was monitoring unauthorized activity on port 80. Port number 5050 was chosen for this purpose, as almost all other ports were filtered. Also, he found a simple dynamic DNS setup, and so the original URL of the Judge was http://pegjudge.ath.cx:5050/. The Judge at that time was very new; it had a very basic set of features, but it got the job done. Guru wrote most of the back end, and Hanson wrote most of the front end. The core grader, responsible for setting up a sandbox and measuring resource usage, was taken from Mooshak, as no PEG member at the time had the expertise in Unix-based systems necessary to write a grader from scratch.

Development

The Judge saw a dramatic improvement in the interface over the course of the 2008/2009 school year as Hanson completely overhauled the front end, leaving the interface cleaner, crisper, and more professional-looking. He also overhauled the back end, switching from the ad hoc PHP functions being used before to the secure and high-performance CodeIgniter framework. (The grader, however, was only tweaked slightly.)

By the end of that year the Judge in its current form was essentially complete, a good thing as Hanson was off to university and would likely have other things on his mind. It even included support for running contests and posting comments. Brian, who had been helping with administration since the beginning of the year, took over at this point as primary administrator of the Judge. Brian's primary contributions were the addition of this wiki to the site and support for various additional popular programming languages.

Later years

With the passing of Ms. Plachta, the Judge was never as popular as before, for students were no longer being pushed to earn 1000-odd points per year. But there were still some PEG members using it; Canada’s IOI teams trained on it; and Brian and a friend from Waterloo C.I. ran a mock DWITE contest. The decision was thus made to keep the Judge alive. When the Plachta house was sold in June 2010, Jacob could no longer host the Judge, so the Judge was migrated to Brian's laptop; this hosting was again illegitimate. When Brian left for university at the beginning of September 2010, he could not risk violating the residence contract that stated that hosting servers was strictly disallowed, so the Judge was left without hosting for a few weeks. Brian, Jacob, and Hanson were hopeful about convincing the Center for Education in Mathematics and Computing at the University of Waterloo to grant PEG hosting, for this is the organization primarily involved with high achievement in computer science at the high school level, running the Canadian Computing Competition and sending Team Canada to the IOI. Almost all problems from CCCs of years past, both stage 1 and stage 2, were already in the PEG Judge's archive, the interface had taken two years of work, the grader had been tweaked precisely, and, all in all, the PEG Judge was the first in Canada with a level of sophistication on par with those of the major online judges of the time. This did not work out, so Brian started looking for hosting companies.

Recent hosting

The Judge was migrated a second time at the end of September 2010 to a OpenVZ box hosted by IntoVPS, which offered 512 MB guaranteed RAM, 30 GB disk space, 500 GB monthly bandwidth, and a quad-core CPU for only $10 USD/month. The :5050 suffix on the Judge URL was dropped, and the web server simply bound to port 80, since the hosting was then legitimate; Jacob also acquired the wcipeg.com domain from James Rosselet (a former PEG member), giving the URL of the Judge as the current http://wcipeg.com/.

A month later, it was again migrated to hosting by Virpus, which offered 1 GB guaranteed RAM, 50 GB disk space, 1600 GB monthly bandwidth, and six CPU cores, as Brian thought this was a better plan.

However, Virpus hosting suffered from too much downtime, so in June of 2011 the Judge was migrated a fourth time to a VolumeDrive box.

In October of 2011, the Judge was migrated a fifth time to hosting offered by LoveVPS, this time a Xen box, due to Brian's dissatisfaction with VolumeDrive's support and the incomplete virtualization of OpenVZ containers in general.

Although LoveVPS was a satisfactory VPS host, Brian was again forced to migrate the Judge in May 2013, this time to HostingBeam (then PhoenixVPS); the reasons for this move will not be discussed here. HostingBeam suffered from strange performance issues: the virtualized file system was very slow, and so was ptrace, an important operating system API used for automatic judging. These issues could be temporarily fixed by rebooting, but it would have been necessary to reboot the VPS more than once a week in order to maintain satisfactory performance.

Brian ultimately could not diagnose these issues with the HostingBeam VPS, so around June 2014, the Judge was moved to a Linode plan with 2 GB of RAM. (It is worth noting that Linode is generally considered one of the best VPS hosts, but was much more expensive around 2010; it became more feasible to host an online judge on Linode after April 2014, when the company doubled the RAM on all of their plans.)

Future plans

Brian plans to rewrite the core grader from scratch, to improve support for languages such as Java that invariably use multiple processes (something that the Unix API does not handle well). He also plans to refactor and then open-source the Judge code. It's currently uncertain whether either of these projects will ever see the light of day.

Brian also hopes that Google Fiber will one day come to Silicon Valley, where he currently resides, so that he can host the Judge on a dedicated server in his own home.

Retrieved from "http://wcipeg.com/wiki/index.php?title=Judge:History&oldid=1834"