A note on our anonymous judging process, by Board member Roger Martin
18 October, 2020
A key measure of sophistication in a field is one’s ability to make fine distinctions in qualitative properties important to that field. For example, we know a Sommelier is good if she can make a distinction between, say, the taste profiles of a 1985 versus 1986 Bordeaux – two very good years but each with a distinctive taste profile. Meanwhile, the best a novice wine drinker may be able to do is distinguish between, say, a random Bordeaux versus Burgundy.
However, we also know that the mind regularly overrides the veracity of these distinctions driven by bias unrelated to the features being judged. For example, if the Sommelier is asked to compare a Château Lafite-Rothschild to a Château Batailley, she is likely to find it hard to discipline her taste buds to forget that in 1855 when the Bordeaux classification system was established, the former was declared a First Growth prince and the latter a Fifth Growth pauper – forever – even though their vineyards are a mere 5 miles apart. That is why, when serious wine tasting happens, it is done blind – to help the tasters let their true skills work unfettered by the bias that is difficult if not impossible to suppress.
While the privilege enjoyed by Lafite-Rothschild may seem trivial – the slight is to an inanimate bottle of Batailley – it does affect the real people cultivating the vines, harvesting the grapes, making the wine and distributing the finished Batailley. After all, if you want a bottle of the brilliant 2016 vintage, the Lafite currently sells for US$825 while the Batailley a mere $67. In blind tests, Lafite certainly scores better (97 points out of 100). But the Batailley scores an also brilliant 93 points – a difference in quality that would require the Sommelier above to discern. Nonetheless the princely Lafite trades for 12 times more.
There are, of course, far bigger issues of bias and privilege in the world than with Bordeaux wineries and when it came to judging our important prize, the Nine Dots Board wanted to be a positive example. So, we made the decision to judge the Nine Dots Prize blind with the hope of letting us use our full ability to make unfettered distinctions in the quality of the submissions. Using a blind process was by no means a complete first. It has been done famously for orchestra auditions, and less famously for the allocation of Hubble telescope time slots and the evaluation of software code. But it has not, to our knowledge, been done in a literary competition like this.
It does take some intestinal fortitude. After the fact, the Nine Dot winners could be revealed to have had an embarrassing past or to be a fraud of some sort. But if you believe in your ability to make distinctions, you must maintain confidence. We are, of course, thrilled with the first two winners and gratified that one is an Asian woman from the global south and the other is a white man from the global north.
But there was a moment when our resolve was tested. In the first competition, we were down to two finalists, each of whom wrote a submission the credibility of which hinged an important facet of their personal work experience – in the case of our eventual winner, James Williams, it was time inside the proverbial beast at Google. As a Board, we struggled with the question of what would happen if we found out, after the fact, that the winner actually hadn’t had the work experience that was integral to the winner’s argumentation. It would have been a serious embarrassment if Williams had turned out to have never worked a minute at Google!
But as always, if there is a will there is a way. We wanted to both protect the integrity of the Nine Dots Prize and maintain the blindness of the review process. After much discussion, a resolution emerged: we would ask Senior Prize Manager Jane Tinkler, who was privy to everything in the finalists’ files but not involved in the shortlisting or judging process, not to reveal anything about their identities but to answer one single question: From her knowledge of the finalists’ resumes, did each actually have the critical work experience claimed in the submission? Her one-word answer for both was: Yes! She confirmed the legitimacy of both implicit work claims without revealing another thing about either candidate, enabling us to feel confident while making our decision as free of bias as possible. It can be done – but only if you are committed to the principle!
Roger Martin