Back to [Archive] Meta

1
Meta-selection: selecting the selection criteria for appointing reviewers
There should be clear criteria on how future reviewers will be selected for OP. I propose a discussion where everyone can contribute their ideas, which will be followed by a vote. My proposal is this:
There should be both general and specific criteria. General criteria are relative to the appointment of a person as reviewer. Specific criteria refer to requirements that a reviewer must meet in order to be eligible to review a specific paper.

General requirements:
Scientific expertise: Referees must have demonstrated expertise in the key topics of the study presented and/or the methods used. They must have published as lead authors at least 5 articles in peer-review journals. Referees cannot have co-authored a paper with the reviewer in the last 2 years, as this is a less generic and more objective criteria than being a "close collaborator".

Author/publication specific requirement:
Independence: Referees must not be working at the same institute as the authors, should not be close collaborators of the authors or in other ways personally, financially or professionally associated with them. Referees must declare any conflicts of interest on the published report.
Referees cannot have co-authored a paper with the reviewer in the last 2 years.
Admin
As mentioned before, these requirements cannot be met currently.

If we had 20 reviewers, I would not have strong problems with this. It would lead to strategic co-authorship among reviewers though. I.e. that authors will avoid co-authoring with each other in order to avoid ineligibility of reviewing each others' papers.
As mentioned before, these requirements cannot be met currently.

If we had 20 reviewers, I would not have strong problems with this. It would lead to strategic co-authorship among reviewers though. I.e. that authors will avoid co-authoring with each other in order to avoid ineligibility of reviewing each others' papers.

You keep attempting to veto this proposal when you have no right to do so and this is not helping the discussion at all. I suggest you stop doing this.Step aside and let others discuss this.
I want to know what other reviewers think, not just your opinion. Besides I already explained that the general requirement is not retrospective and you keep ignoring it. It is relative to selection of reviewers in the future. And I totally disagree with your idea about strategic co-authorship. It is better to avoid conflict of interest between authors and reviewers. What do others think? I think there should be objective requirements for appointing reviewers.
Admin
I don't know why you would disagree about the possibility of strategic authoring. It is well-known from studies of voting systems that one must take into account strategic voting behavior. It is no different with authorship.

http://en.wikipedia.org/wiki/Tactical_voting
I don't know why you would disagree about the possibility of strategic authoring. It is well-known from studies of voting systems that one must take into account strategic voting behavior. It is no different with authorship.

http://en.wikipedia.org/wiki/Tactical_voting


There is a big difference: for each political system there is only one possible electoral outcome. With journals, there are many and none is forced to publish on OP rather than f1000 or Elsevier, it's up to the author's choice. Other journals adopt the same policy so those that avoid one reviewer in another journal can publish on OP and those that need to avoid a reviewer on OP can publish in a different journal. Your argument would be valid only if OP were the sole open access journal, but there are many others and it's better to allow free competition rather than a protectionist system where each journal limits review quality in order to be an isolate planet. Authors can invite one external reviewer of their own choosing but they can also invite as many other reviewers as they want, provided they pass the vote of the current OP reviewers. But this vote needs to be based on clear requirements or it will be arbitrary.
Morever, the benefits that would accrue to review from a system of no co-authorship between author and reviewer, are bigger (impartiality, no conflicts of interests)so are likely to outweigh any other possible costs. A judge cannot be the culprit's attorney.
Can someone other than Emil have their say on my reviewer's policies proposal?
I have to agree with Duxide. The proposed criteria are so stringent that we would have trouble finding reviewers who meet all of the criteria. We might also miss out on some interesting candidates who meet some of the criteria but not all.

It might be better to settle for something like the following:

- reviewers must have a graduate degree or are working toward completion of a graduate degree in a relevant field of study (psychology, anthropology, genetics, evolutionary biology).

- reviewers must have published at least two articles in a peer-reviewed journal

- reviewers must abstain from reviewing manuscripts where there may be a conflict of interest, i.e., a recent history of academic collaboration with the author or authors.

- reviewers must disclose their real name and contact information to the editorial board. In any case, I don't see how we can evaluate a prospective reviewer unless we know his or her identity.

Frankly, I'm bothered by the prospect of having low standards for existing reviewers and high standards for future reviewers. Shouldn't we have a mechanism for removing existing reviewers who don't do their job or who act in a disreputable manner? By "disreputable" I mean:
- not engaging with contrary views and simply repeating one's own views
- using abusive language
- not reviewing manuscripts (which is presumably a reviewer's main function)
I have to agree with Duxide. The proposed criteria are so stringent that we would have trouble finding reviewers who meet all of the criteria. We might also miss out on some interesting candidates who meet some of the criteria but not all.

It might be better to settle for something like the following:

- reviewers must have a graduate degree or are working toward completion of a graduate degree in a relevant field of study (psychology, anthropology, genetics, evolutionary biology).

- reviewers must have published at least two articles in a peer-reviewed journal

- reviewers must abstain from reviewing manuscripts where there may be a conflict of interest, i.e., a recent history of academic collaboration with the author or authors.

- reviewers must disclose their real name and contact information to the editorial board. In any case, I don't see how we can evaluate a prospective reviewer unless we know his or her identity.

Frankly, I'm bothered by the prospect of having low standards for existing reviewers and high standards for future reviewers. Shouldn't we have a mechanism for removing existing reviewers who don't do their job or who act in a disreputable manner? By "disreputable" I mean:
- not engaging with contrary views and simply repeating one's own views
- using abusive language
- not reviewing manuscripts (which is presumably a reviewer's main function)


Thanks for your suggestions. I agree with n.2 and 3 but not with n.1 as I do not believe in credentialism.
About removing reviewers who act in a disreputable manner, it's subjective to judge as everyone has a different opinion of what is disreputable. I do not think that reviewers can be banned simply because they're not reviewing manuscripts for 2 reasons: 1) A reviewer is doing no harm if he does not review a paper and 2)OP is based on the idea that the editor cannot choose the reviewers for each paper (because this would introduce bias), rather it's the reviewers who decide which paper they will review, so your proposed criteria would infringe this principle.
It's also difficult to judge when a reviewer is "not engaging with contrary views and simply repeating one's own views". Abusive language is discouraged but I think there are many different ways to offend someone other than abusive language. Of course racist or personal attacks will not be tolerated.
Admin
Peter,

Frankly, I'm bothered by the prospect of having low standards for existing reviewers and high standards for future reviewers. Shouldn't we have a mechanism for removing existing reviewers who don't do their job or who act in a disreputable manner? By "disreputable" I mean:
- not engaging with contrary views and simply repeating one's own views
- using abusive language
- not reviewing manuscripts (which is presumably a reviewer's main function)


The current review team may vote to remove any current reviewer.
It's not very encouraging if a reviewer needs to have published x number of papers in respectable peer-reviewed journals. I think blog articles and websites are just as fine.

I agree with all the talk about conflict of interests. But not with the fact that the author can choose a reviewer external to the journal. He can, if he wants, choose someone, but only the ones in the current lists of ODP/OBG reviewers.

As for credentialism, at first glance it might seem reasonable, but even myself I believe I can do as much as a good number of scientists who are publishing in psychology and yet I have no degree on the relevant fields (I'm not even a scientist at all). If this is what it takes to be a reviewer here, probably a lot of people would be dismissed as future candidates, some of the existing reviewers (myself) should be removed from the current list. It's not that I do not wish to be removed from the list, but I just wanted to say that with such a system, you won't get any new reviewers.

Concerning the expertise in the key topics as a condition for being a reviewer for any given OP papers, what do you have in mind ? That the reviewers must have published several papers (or blog articles) on these key topics ?

Frankly, I'm bothered by the prospect of having low standards for existing reviewers and high standards for future reviewers. Shouldn't we have a mechanism for removing existing reviewers who don't do their job or who act in a disreputable manner?


I would agree with both. But I tend to think that the low standards for existing reviewers is due to the fact that you can't get any reviewers by starting with having high standards. That does not mean that the existing reviewers should not try to get approvals; for instance, despite being actually OBG reviewer, I'm planning to write several blog articles (and perhaps papers) and submit them for being approved by the current OBG reviewers (Duxide, Frost, Chuck, and eventually Meisenberg and Kura if they have the time to do it...). But that may take some time until I finish everything.
Admin
Like Piffer above, I am staunchly opposed to credentialism. The current review team is well-qualified to judge who is educated about a field, and there is no need to rely on a proxy such as having a degree from some official institution. I also think that credentialism goes against the very spirit of science. Traditionally, science has been practiced by a very wide variety of people, many of them doing it as a hobby. This has been seriously hampered by the practice of extremely pricey journal subscriptions in recent decades. The open access (and open science by extension), which I consider myself a part of, is attempting to reverse this undesirable state of affairs.

If some kind of formal requirement must be made, it should be a standardized test of relevant knowledge. But in most cases, it will suffice simply to read the person's writings on e.g. a blog. This is how Piffer and I selected the initial review team. Primarily based on writings on Human Varieties.

Disclaimer: I have no academic degree and I'm not pursuing a relevant one either (I am a linguistics student).
"If some kind of formal requirement must be made, it should be a standardized test of relevant knowledge."

Fine. That would be an acceptable alternative, although a test would not measure things like professional integrity. It's possible to be very intelligent and also very dishonest. Those two qualities are not exclusive.

In my opinion, our reviewers should be reviewed from time to time. If some of them are not doing their primary job (reviewing manuscripts), while using this forum as a megaphone for their views, they should lose their status as reviewer.
"If some kind of formal requirement must be made, it should be a standardized test of relevant knowledge."

Fine. That would be an acceptable alternative, although a test would not measure things like professional integrity. It's possible to be very intelligent and also very dishonest. Those two qualities are not exclusive.

In my opinion, our reviewers should be reviewed from time to time. If some of them are not doing their primary job (reviewing manuscripts), while using this forum as a megaphone for their views, they should lose their status as reviewer.


I think a more reasonable requirement should be that a reviewer has been published on international peer-reviewed journals.
I am not sure about punishing those that are not reviewing manuscripts, as I do not see what harm they do in not doing so and they're not paid to do this but they do it using their unpaid free time.
"I am not sure about punishing those that are not reviewing manuscripts, as I do not see what harm they do "

The harm is analogous to that of the Russian and Nigerian spammers we have to block from time to time. Our forum provides some people with a free megaphone. It's free advertising.

I agree that reviewing is a thankless and often onerous job. I'm simply saying that we should remove "reviewers" who have not done any reviewing and who seem uninclined to do any. I don't see the difference between that kind of person and a spammer.
"I am not sure about punishing those that are not reviewing manuscripts, as I do not see what harm they do "

The harm is analogous to that of the Russian and Nigerian spammers we have to block from time to time. Our forum provides some people with a free megaphone. It's free advertising.

I agree that reviewing is a thankless and often onerous job. I'm simply saying that we should remove "reviewers" who have not done any reviewing and who seem uninclined to do any. I don't see the difference between that kind of person and a spammer.


I do not think they're spammers. Reviewers are simply people who have passed our screening and meet the requirements to review papers. I do not think this forum is free advertising. It's simply a place where people can exchange ideas and promote their own views.
1