In any case, how can we honestly pretend to be scientists if we don’t even believe in the principle of double-blind. Computer Science still has some ways to go in this respect: e.g. I don’t know of any research group who does his benchmarks using double-blind procedures.

Basically, the point is not that it has to be impossible for reviewers to guess the author(s) in order to ensure fairness, rather that some uncertainty is more conducive to fairness than none. After all, you can sometimes guess who wrote a review as well, but this is not often cited as an argument that blind review is pointless.

I didn’t find it that annoying to (make some effort to) disguise myself, but YMMV.

“Additionally, I have heard that in other sub-disciplines of computer science where double-blind reviewing is the norm and stricter procedures are used, that this has resulted in less honest reviewers stealing research because the author was unable to make their results public at the time of review.”

That’s not a problem here, because of the specific setup:

Our job is to evaluate your submission and write reviews without trying to discover who you are. For example, Google may know who has written what, but we will resist the temptation to ask!
…
Once all reviews are written, your identity will be revealed to the program committee. This procedure makes it possible for the committee to understand the entire context of your work as well as to perform such necessary tasks as checking for dual submissions.

“Finally, the reviewing process tends to be so random I would be really surprised if whatever slight unconscious bias people have isn’t vastly outweighed by other factors. If someone has actually done a study on the statistical impact in a realistic reviewing setting I would be interesting in a pointer to it. For example, do any of these studies consider the multi-author case? Was there a program committee meeting?”

I don’t know of anything that specific, but the gender gap in e.g. the Sweden fellowship study is large enough that I’m not optimistic about conference reviewing. Also, it’s not enough to argue that the effect of the bias ought to be less than other factors, including but not limited to dumb luck—that may be true on a per-submission basis, but it’s the effect over entire careers that matters. That’s one of the arguments made here.

If anything, I would be more concerned about why so few women choose the sub-discipline of programming languages. I’m fairly sure no one has shown that it is biased reviewing driving them away to study machine learning or robotics, for instance.

I don’t know anything specifically about ICFP. But one need not run a study about every conference in every field to expect that, with high probability, the reviewers will exhibit the same unconscious bias that’s been shown repeatedly in other contexts. A study on postdoctoral fellowship evaluations in Sweden, for example, showed a large gender gap between evaluators’ estimate of “scientific competence” for applicants with similar publication records (measured by “impact scores”). And there are other studies where people evaluating papers that were identical except for the author’s name (obviously male, indeterminate due to using initials, and obviously female) rated the “female” papers lower, on average, than the others.

(Both men and women have this obviously unconscious bias—so just getting more women on program committees won’t do it.)

I assert this based upon the fact that when ranking authors by the number of accepted ICFP papers, a certain female researcher I know is very highly ranked. The number one position is, not surprisingly, safely held by Simon Peyton Jones. I suppose one could quibble about the fact that in many instances she was not the sole author. I’ll have to e-mail my colleague to get the precise data, but it would be worthwhile to see.

More concretely, if I see [REDACTED 17]’s name on a paper I’m going to expect that their prose will be awkward and their results not particularly compelling—because that’s how their papers seem to be. I don’t even know if that hurts them (I expect it to be mediocre so I’ll think it is, regardless of its merits) or helps them (if it’s actually a great paper, I’ll be pleasantly surprised and rate it even higher than it deserves), but it seems like it’s got to have some effect.

There are also some highly disturbing studies about the effects of female authorship on the evaluation of a paper; if double-blind mitigates that at all, I’d say it’s worth trying.