Justice is not double-blind
This year ICFP chose to try double-blind reviewing. When I first heard about it, I felt that it would be hopeless to try to anonymize submissions. When I was asked to be a sub-reviewer on a paper today I was proven correct. I only had to read the first line of the abstract to before I knew with certainty at least one of the authors of the paper.
Will this affect my review? No. I think highly of their work, but that isn't going to stop me from giving it critical scrutiny. Maybe if I were given just a minute to choose between this paper and another for acceptance, knowing the author(s) might bias my selection, but I expect most reviewers are going to be responsible enough to spend far longer than that considering the merits of the papers they were assigned.
The whole process just seems like it generates busywork for everyone from the authors to the program chair, and it seems unlikely that enough data could be collected to show that it had a measurable impact on the selected program. I would be curious as to whether a non-trivial number of authors even perceive the process as being more impartial than usual.
Joshua Dunfield said,
April 20, 2007 @ 1:29 am
I’ve thought about this a bit. I think knowing from the first sentence of the abstract is unusual. At the end of most abstracts I might have a good guess about who some of the authors are, but I don’t think I’d know, and my expectations wouldn’t be as definitively altered as they might be if I actually saw Famously Good Writer/Researcher X’s name or, contrariwise, Lifelong Betrayer of the Fatherland Y’s name right below the title.
More concretely, if I see [REDACTED 17]’s name on a paper I’m going to expect that their prose will be awkward and their results not particularly compelling—because that’s how their papers seem to be. I don’t even know if that hurts them (I expect it to be mediocre so I’ll think it is, regardless of its merits) or helps them (if it’s actually a great paper, I’ll be pleasantly surprised and rate it even higher than it deserves), but it seems like it’s got to have some effect.
There are also some highly disturbing studies about the effects of female authorship on the evaluation of a paper; if double-blind mitigates that at all, I’d say it’s worth trying.
washburn said,
April 20, 2007 @ 6:57 am
I have to admit that I had not thought about the female authorship issues. However, in the case of ICFP, I don’t believe that is a problem. If you’ve heard otherwise I would be interested to know.
I assert this based upon the fact that when ranking authors by the number of accepted ICFP papers, a certain female researcher I know is very highly ranked. The number one position is, not surprisingly, safely held by Simon Peyton Jones. I suppose one could quibble about the fact that in many instances she was not the sole author. I’ll have to e-mail my colleague to get the precise data, but it would be worthwhile to see.
Joshua Dunfield said,
April 20, 2007 @ 6:46 pm
The existence of one very ICFP-successful woman doesn’t prove anything, except that the effect of bias isn’t utterly overwhelming in all instances.
I don’t know anything specifically about ICFP. But one need not run a study about every conference in every field to expect that, with high probability, the reviewers will exhibit the same unconscious bias that’s been shown repeatedly in other contexts. A study on postdoctoral fellowship evaluations in Sweden, for example, showed a large gender gap between evaluators’ estimate of “scientific competence” for applicants with similar publication records (measured by “impact scores”). And there are other studies where people evaluating papers that were identical except for the author’s name (obviously male, indeterminate due to using initials, and obviously female) rated the “female” papers lower, on average, than the others.
(Both men and women have this obviously unconscious bias—so just getting more women on program committees won’t do it.)
washburn said,
April 20, 2007 @ 8:03 pm
Much of my original point was that double-blind reviewing is a farce unless the authors invest a significant amount of time (the busy work) trying to disguise themselves. Additionally, I have heard that in other sub-disciplines of computer science where double-blind reviewing is the norm and stricter procedures are used, that this has resulted in less honest reviewers stealing research because the author was unable to make their results public at the time of review. Finally, the reviewing process tends to be so random I would be really surprised if whatever slight unconscious bias people have isn’t vastly outweighed by other factors. If someone has actually done a study on the statistical impact in a realistic reviewing setting I would be interesting in a pointer to it. For example, do any of these studies consider the multi-author case? Was there a program committee meeting?
If anything, I would be more concerned about why so few women choose the sub-discipline of programming languages. I’m fairly sure no one has shown that it is biased reviewing driving them away to study machine learning or robotics, for instance.
Joshua Dunfield said,
April 21, 2007 @ 10:44 am
“Much of my original point was that double-blind reviewing is a farce unless the authors invest a significant amount of time (the busy work) trying to disguise themselves.”
I didn’t find it that annoying to (make some effort to) disguise myself, but YMMV.
“Additionally, I have heard that in other sub-disciplines of computer science where double-blind reviewing is the norm and stricter procedures are used, that this has resulted in less honest reviewers stealing research because the author was unable to make their results public at the time of review.”
That’s not a problem here, because of the specific setup:
Our job is to evaluate your submission and write reviews without trying to discover who you are. For example, Google may know who has written what, but we will resist the temptation to ask!
…
Once all reviews are written, your identity will be revealed to the program committee. This procedure makes it possible for the committee to understand the entire context of your work as well as to perform such necessary tasks as checking for dual submissions.
“Finally, the reviewing process tends to be so random I would be really surprised if whatever slight unconscious bias people have isn’t vastly outweighed by other factors. If someone has actually done a study on the statistical impact in a realistic reviewing setting I would be interesting in a pointer to it. For example, do any of these studies consider the multi-author case? Was there a program committee meeting?”
I don’t know of anything that specific, but the gender gap in e.g. the Sweden fellowship study is large enough that I’m not optimistic about conference reviewing. Also, it’s not enough to argue that the effect of the bias ought to be less than other factors, including but not limited to dumb luck—that may be true on a per-submission basis, but it’s the effect over entire careers that matters. That’s one of the arguments made here.
James Cheney said,
May 1, 2007 @ 5:26 am
Transactions on Database Systems (TODS) recently instituted double-blind reviewing; a thorough discussion of the pros and cons is here.
Basically, the point is not that it has to be impossible for reviewers to guess the author(s) in order to ensure fairness, rather that some uncertainty is more conducive to fairness than none. After all, you can sometimes guess who wrote a review as well, but this is not often cited as an argument that blind review is pointless.
Stefan Monnier said,
May 9, 2007 @ 4:10 pm
While I completely agree that I can often tell a large part of the academic genealogy of the authors just by reading the abstract and/or looking at the notations they use, I think that only works for regular contributors. If you’re presented with a work which doesn’t come from someone closely related to one of the main research groups, you may very well guess wrong. Even more so if you really do not know the author (in which case you’d have to guess that you don’t know him).
In any case, how can we honestly pretend to be scientists if we don’t even believe in the principle of double-blind. Computer Science still has some ways to go in this respect: e.g. I don’t know of any research group who does his benchmarks using double-blind procedures.