Project:Judge Article of the Year

From Judge Wiki (English)
Jump to: navigation, search


Project lead: Christian Gawrilowicz
Start: 2005
Status: ongoing (yearly)

Project description

The "Judge Article of the Year" is a yearly project that chooses and awards the (at least in the eyes of the voters) best article that was written about judging matters in the year before. The voting process and the eligible articles varied with the years, details can be found here.

Project staff

Documentation

Project report 2012

Abstract
This report describes the voting process for the Judge Article of Year 2012.

Preliminary work
As last year all judges were allowed to participate with their articles on various websites. Writers of regular columns (i.e. Blackborder …) were asked to submit one of their articles for the contest. All eligible articles were assigned to one of four categories ("Education & Development", "Report", "Rules & Regulations", "Work & Run a tournament"). In total there were 7, 8, 14, and 7 articles in the categories. This process was done in January 2012. Based on the experiences of the 2012 contest, it was decided that Surveymonkey should not be used for the voting. As the voting functionality only allows five answers per question and all categories had more entries than five, various options were considered. In the end it was decided, that all L3+ receive two votes for each category and should send in their votes via email. The final voting (one winner of each category) should take place on DCIX and be open for all judges.

The voting process
The category voting, open for L3+ judges, was originally scheduled to last until mid-February, but this collided with a Pro Tour where many L3+s were occupied, so the voting deadline was moved to the end of February. For about three weeks the start of the final voting was announced on various channels (Facebook, Twitter, JudgeApps …), announcing the winner of each category (including links to the winning articles). The final voting (five entries – one category was tied) took place as announced on DCIX and lasted for three weeks until March, 17. 153 people participated.

Lessons learned
The final voting worked out as planned and the participation was satisfying. What needs to be reconsidered is the method for the preliminary votings, especially as it’s still unclear if and when a voting functionality will be available on JudgeApps. The idea to have a voting committee to do this preselection that was raised in 2012, would be one of the options. Allowing the L3+s to preselect the articles for a later stage is a feasible idea, if their vote should be the only one before the final one needs to be discussed.


Project report 2011

Abstract
This report describes the Judge Article of the Year 2011 voting process.

Preliminary work
For the voting of the best article of the year 2011 all judges were allowed to participate with their articles on various websites. As in 2011 the writers of regular columns were asked to choose and submit one article to the voting process. The same four categories as for the article of the year 2010 were used – "Education & Development" (6 articles), "Report" (7), "Rules & Regulations" (4), "Work & Run a Tournament" (13), and all the articles in the Judge Article Index were put in one of the categories. This process was done by the end of 2011. As the voting functionality of DCIF and DCIX were no longer available, and it took too long to implement a new one for IMJ, an external tool (Surveymonkey) was chosen to host the voting process.

The voting process
For four weeks a vote for one category was started, that lasted two weeks. The voting was promoted with posts to the Judge Mailing list and on the "MagicJudges" Facebook account. The four category winners were pitted against each other in a final vote that lasted only one week.

Lessons learned
As an external vendor was used for the voting theoretically everyone could vote. Which obviously happened when the link to the votings were published on Facebook. For the future an internal solution would be better. It might also be better to restrict the voting for the categories to a certain group of people (L3+ for example or a selected committee) – the final voting should always be open for all judges. If an external vendor is used again, then the link to the voting should not be published outside the Judge Mailing list to avoid skewed results. (In one category the winner received more than 200 votes – compared to about 15 to 20 for all other participants in this category). What worked well was the posting of the winner's name and a link to his/her article. – This raised the awareness for judge articles and should be repeated in the future.


Project report 2010

Abstract
This is a summary of the project "Judge Article of the Year 2010". It gives an overview what was done in this project, some explanations, lessons learned and possible improvements for the following years.

Project description
The purpose of the JAotY is to determine the "best" judge article written on the internet. The method used to determine the winner was a series of votes – first within defined categories and after this the category winners to determine the overall winner. All articles were eligible, provided they were written by a judge and were about a topic interesting for judges, even if the target audience were players.

Preliminary work
The first step was to decide how the voting process should be organized. A short discussion between Carlos Ho and Christian Gawrilowicz lead to the result, that similar to the process used the year before there should first be votes within defined categories and after this the winners of these would be placed into a final vote. Much thought and effort was put into finding suitable categories in the year before, and it was decided that the same (Report, Work & Run a Tournament, Rules & Regulations, Education & Development) will be used again, as they proved useful.

The next step was to decide which articles are eligible for voting. The base for this decision was the Judge Article Index (maintained by Philip Schulz) in the Judge Wiki. On this page title, author, date written, web site, link and sometimes keywords for all judge articles for the last couple of years are listed.

From the beginning on all articles that were written by (at this time) Level 4+ judges were declared ineligible with the exception when a L4+ was a co-writer. Another issue that was raised and discussed was regular judge columns on pages like StarCityGames, BlackBorder and others. It was argued that these judges already receive lots of recognition for their work there and as about 80 articles were published in 2010 some cuts had to be made. After some discussion it was decided that each writer for a regular column could nominate one of his/her articles to be included in the voting. At this time it was pointed out that Q&A columns like "Cranial Insertion" and others were missing from the Judge Article Index and should be included. Some participants in the ensuing discussion agreed, some disagreed, in the end it wasn't settled, although it ultimately is the decision of Philip Schulz as the owner of the Judge Article Index what to include and what not.

An alternative approach to cut down the number of articles was to have two separate votes – one with "official" articles on Wizards.com and DCIF and the other with articles on commercial sites. This idea was soon discarded, as it would water down the idea of having "the best" article of one year.

Parallel to this discussion all the articles of 2010 received their own Wiki page with meta information and all were put into one of the four categories mentioned above. Unfortunately this took longer then planned, so the voting could only be started at the end of March.

The voting process
The votings for the categories were set up in DCIF. Each lasted two weeks, were announced on the Judge Mailing list and hat staggered starting times, so that after one week there were always two votings active at the same time. The category winners were posted to the mailing list together with an announcement of the next vote. All the nominated articles, including winners were also put on a Wiki page. The final vote started when the fourth category voting ended and due to a mistake lasted three weeks instead the usual two. Participation was decent – per category voting about 35 people, for the final vote about 80 people.

Lessons learned
The four categories defined in 2010 are still valid und useful and should be used for all future JAotY votings. Two weeks per category voting together with overlapping durations seemed to have worked well.

Potential improvements
The voting process should be started much earlier in the year (preferably mid-January). Having a winner of the year 2010 in May 2011 is a bit late. For this the categorizing of the articles has to be finished earlier. About two months before the voting starts some announcements to the mailing list should be made, "selling" the whole vote to increase participation numbers (and to make more judges aware of the resource "Judge Article Index").

Summary
The process could be started earlier, but still went quite smoothly, and had an unexpected winner (normally articles in the Report category are huge favorites). For future years some more emphasis should be put on having the categorizing finished earlier to be able to start the vote by mid-January, end of January at the latest.