Jun 20, 2023·edited Jun 20, 2023Liked by Helen Pluckrose
Excellent reading as usual. I had never thought what you, James, and Peter did / found discredited (or aimed to discredit) the scientific peer review process. It highlighted serious issues (namely ideological capture) in those fields / journals where you submitted papers.
(Of course there may be other serious issues in other fields. And I am worried about new EDI/DEI initiatives in academic publishing more generally. Last time I submitted and was asked to review a paper, I was invited to provide personal information about my race and ethnic origin and a bunch of other personal factors. I declined. I must say that being asked this did not instil confidence in me about the fairness of the process in light of the current drive for "diversity". It seems the goal may be to achieve "equity" on editorial boards now. I am worried about this because I am all for any real discrimination being tackled where it exists but I have no desire to be subjected to positive discrimination to try and make numbers look good. This would devalue everyone and is not good for science. But this is separate from the point you are making).
Please write more often if you can Helen. Your voice is greatly missed! Maybe you tweet, I don't know... not a Twitter user here. This long-form stuff is much better anyway. :-)
I think the biggest problem with peer review is its **mythology**
Nobody in my field (physics) thinks peer review is anything other than a reasonable 'check' to see if any (relatively) obvious errors can be spotted etc. Passing peer review is basically a statement that "2 or more people looked at it and thought it was OK and made sense and couldn't spot any crazy errors"
There are just too many examples of stuff that's passed peer review that turn out to be incorrect upon closer analysis. Heck, I've even written a paper that contained a stupid error that got published (I solved a differential equation using a flawed method. The solution I got was actually correct - because with a differential equation you can actually check your solution - but the whole approach I used was embarrassingly dumb. None of the reviewers spotted it)
The problem is that too many people seem to assign a much greater significance to the process than it warrants; the mythology seems to be along the lines of "it has passed peer review and *therefore* must be correct".
But, as you point out, this is in a field where objective correctness is valued. The review yardstick used in CSJ 'research' seems to be more along the lines of "it agrees with my ideology and so must be correct". This is, perhaps, an inevitable consequence of a 'discipline' that eschews any notion of objective truth from the outset. So, for CSJ papers you're hit with a kind of double whammy; not only is the yardstick a bit wonky, but the label "peer review" itself lends this extra unwarranted degree of mythological credence.
We all know (us scientists) that the peer review process is not perfect, but it is valuable nevertheless. It's (a) a necessary sanity check and (b) often very useful if you get a good reviewer who makes great suggestions for improvement.
The other problem with peer review - and this has been painfully obvious during the covid era - is that the whole process is woefully subject to manipulation. There have been so many examples of (covid) papers that have been rejected for nothing more than the 'crime' of going against the prevailing narrative, and for no clear scientific reason.
You can even see this effect in the published papers themselves. There are many cases where the actual data points to an essentially *opposite* conclusion to the ones given in the abstract and summary. Whilst it's probably wrong to impugn motive, it looks as if the authors adapted their conclusions to *appear* supportive in order to be able to publish.
I think this kind of almost automatic rejection of "off narrative" papers is endemic in the grievance field. You could write the most magnificent, detailed, and brilliant analysis ever committed to paper, but if it came to the conclusion that some systemic "ism" wasn't very significant, what would be the chances of publication in a grievance journal? I would suggest a probability almost indistinguishable from zero.
Answer #1: But you inspected dog genitalia in that park. How was this not a qualitative appraisal of quantitative data?! 🤨
Answer #2: Ah, but Covid is adjacent to Whiteness, and as such has a complex aided by those who've internalised their lymphocyte supremacy. 🤔
Answer #3: I am at times led to believe that a number of those who purport to understand the work you did to be either a little disingenuous in their interpretation, or so deep into "that part" of the anti-Woke/vax cult as to see everything as confirming their bias. I'm unsure who gave them this idea, but it's been incredibly pernicious throughout the pandemic and beyond, and quite frankly a danger to them and to others. I prefer those who, like yourself, try to be persistently liberal as opposed to those who appear to believe their own hype, and for whom only their popularity apparently matters. It goes a long way towards fostering credibility.
Areas of scientific study, specifically within STEM subjects, have typically been far more resilient to fraud and bias than any humanity, and are incredible for their ability to self-correct. As you said, they're not immune to error or fraud, but by Darwin is the scientific method, and peer review within them, harder to fool for any length of time.
Excellent reading as usual. I had never thought what you, James, and Peter did / found discredited (or aimed to discredit) the scientific peer review process. It highlighted serious issues (namely ideological capture) in those fields / journals where you submitted papers.
(Of course there may be other serious issues in other fields. And I am worried about new EDI/DEI initiatives in academic publishing more generally. Last time I submitted and was asked to review a paper, I was invited to provide personal information about my race and ethnic origin and a bunch of other personal factors. I declined. I must say that being asked this did not instil confidence in me about the fairness of the process in light of the current drive for "diversity". It seems the goal may be to achieve "equity" on editorial boards now. I am worried about this because I am all for any real discrimination being tackled where it exists but I have no desire to be subjected to positive discrimination to try and make numbers look good. This would devalue everyone and is not good for science. But this is separate from the point you are making).
Please write more often if you can Helen. Your voice is greatly missed! Maybe you tweet, I don't know... not a Twitter user here. This long-form stuff is much better anyway. :-)
Helen, You and Lindsay and Peter are heroes. You have made your bed...
Who doesn't want their peers to review their work? Narcissists, is who.
I think the biggest problem with peer review is its **mythology**
Nobody in my field (physics) thinks peer review is anything other than a reasonable 'check' to see if any (relatively) obvious errors can be spotted etc. Passing peer review is basically a statement that "2 or more people looked at it and thought it was OK and made sense and couldn't spot any crazy errors"
There are just too many examples of stuff that's passed peer review that turn out to be incorrect upon closer analysis. Heck, I've even written a paper that contained a stupid error that got published (I solved a differential equation using a flawed method. The solution I got was actually correct - because with a differential equation you can actually check your solution - but the whole approach I used was embarrassingly dumb. None of the reviewers spotted it)
The problem is that too many people seem to assign a much greater significance to the process than it warrants; the mythology seems to be along the lines of "it has passed peer review and *therefore* must be correct".
But, as you point out, this is in a field where objective correctness is valued. The review yardstick used in CSJ 'research' seems to be more along the lines of "it agrees with my ideology and so must be correct". This is, perhaps, an inevitable consequence of a 'discipline' that eschews any notion of objective truth from the outset. So, for CSJ papers you're hit with a kind of double whammy; not only is the yardstick a bit wonky, but the label "peer review" itself lends this extra unwarranted degree of mythological credence.
We all know (us scientists) that the peer review process is not perfect, but it is valuable nevertheless. It's (a) a necessary sanity check and (b) often very useful if you get a good reviewer who makes great suggestions for improvement.
The other problem with peer review - and this has been painfully obvious during the covid era - is that the whole process is woefully subject to manipulation. There have been so many examples of (covid) papers that have been rejected for nothing more than the 'crime' of going against the prevailing narrative, and for no clear scientific reason.
You can even see this effect in the published papers themselves. There are many cases where the actual data points to an essentially *opposite* conclusion to the ones given in the abstract and summary. Whilst it's probably wrong to impugn motive, it looks as if the authors adapted their conclusions to *appear* supportive in order to be able to publish.
I think this kind of almost automatic rejection of "off narrative" papers is endemic in the grievance field. You could write the most magnificent, detailed, and brilliant analysis ever committed to paper, but if it came to the conclusion that some systemic "ism" wasn't very significant, what would be the chances of publication in a grievance journal? I would suggest a probability almost indistinguishable from zero.
Answer #1: But you inspected dog genitalia in that park. How was this not a qualitative appraisal of quantitative data?! 🤨
Answer #2: Ah, but Covid is adjacent to Whiteness, and as such has a complex aided by those who've internalised their lymphocyte supremacy. 🤔
Answer #3: I am at times led to believe that a number of those who purport to understand the work you did to be either a little disingenuous in their interpretation, or so deep into "that part" of the anti-Woke/vax cult as to see everything as confirming their bias. I'm unsure who gave them this idea, but it's been incredibly pernicious throughout the pandemic and beyond, and quite frankly a danger to them and to others. I prefer those who, like yourself, try to be persistently liberal as opposed to those who appear to believe their own hype, and for whom only their popularity apparently matters. It goes a long way towards fostering credibility.
Areas of scientific study, specifically within STEM subjects, have typically been far more resilient to fraud and bias than any humanity, and are incredible for their ability to self-correct. As you said, they're not immune to error or fraud, but by Darwin is the scientific method, and peer review within them, harder to fool for any length of time.