Monday, May 6, 2019

Wisdom of crowds and open, asynchronous peer review

I am very much in favor of preprints and open review, but something I listened to on Planet Money recently gave me some food for thought, along with a recent poll I tweeted about re-reviewing papers. The episode was about wisdom of the crowds, and how magically if you take a large number of non-expert guesses about, say, the weight of an ox, the average comes out pretty close to the actual value. Pretty cool effect!

But something in the podcast caught my ear. They talked about how when they asked some kids, you had to watch out, because once one kid said, say, 300 pounds (wildly inaccurate), then if the other kids heard it, then they would all start saying 300 pounds. Maybe some minor variations, but the point is that they were strongly influenced by that initial guess, rather than just picking something essentially purely random. The thing was that if you had no point of reference, then even a guess provides that point of reference.

Okay, so what does this have to do with peer review? What got me thinking about it was the tweet about re-reviewing a paper you had already seen but for a different journal. I'm like nah not gonna do it because it's a waste of time, but some people said, well, you are now biased. So… in a world where we openly and asynchronously review papers (preprints, postpub, whatever), we would have the same problem that the kids guessing the weight of the cow did: whoever gives the first opinion would potentially strongly influence all subsequent opinions. With conventional peer review, everyone does it blind to the others, and so reviews could be considered more independent samplings (probably dramatically undersampled, but that's another blog post). But imagine someone comments on a preprint with some purported flaw. That narrative is very likely to color subsequent reviews and discussions. I think we've all seen this coloring: take eLife collaborative peer review, or even grant review. Everyone harmonizes their scores, and it's often not an averaging. One could argue that unlike randos on the internet guessing a cow's weight, peer reviewers are all experts. Maybe, but I am somehow not so sure that once we are in the world of experts reviewing what is hopefully a reasonably decent paper that there's much signal beyond noise.

What could we do about this? Well, we could commission someone to hold all the open reviews in confidence and then publish them all at once… oh wait, I think we already have some annoying system for that. I dunno, not really sure, but anyway, was something I was wondering about recently, thoughts welcome.

12 comments:

  1. Interesting post. What I like about open peer review is that it is useful for teaching other reviewers what they might have missed in their own reviews. (Of course, it could also show them the nasty side of peer review, but I rarely see that in the journals that I have served as Editor for.)

    Also, I have a slightly different example of this kind of bias. I was asked to review a review paper that was in one of those online discussion journals. Because the authors would figure out who I was, I chose to be not anonymous. I thought the review paper was worthwhile, but there were many flaws in the content and writing that required extensive revisions. If done right, this would probably occupy many months of the authors' time, so I recommended rejection and resubmission after all the flaws were fixed.

    Friends and colleagues of the coauthors starting posting reviews about how important this paper was to publish, most neglecting my point that it was poorly written and needing substantial work.

    In this case, the open peer-review process allowed a camp to form around support for the manuscript. I don't know if the Editor took these comments into account in the decision.

    ReplyDelete
    Replies
    1. I am very happy today with my family. My name is lisa tommy living in USA, My husband left me for a good 3 years now, and i love him so much, i have been looking for a way to get him back since then. i have tried many options but he did not come back, until i met a friend that darted me to Dr.Jato a spell caster, who helped me to bring back my husband after 2 weeks. Me and my husband are living happily together today, That man is great, you can contact him via email jatolovespell@gmail.com Now i will advice any serious persons that found themselves in this kind of problem to contact him now a fast solution without stress.. He always hello, now i call him my father.contact him now he is always online email (jatolovespell@gmail.com) or contact him on his whatsapp mobile line +2348140033827...

      Delete
    2. Herbal Penis Enlargement product is 100% guarantee to Enlarge and get a better ERECTION ,the reason why most people are finding it difficultto enlarge Penis is because they bedlieve on medicalreport, drugs and medical treatment which is nothelpful for Penis Enlargemaent . Natural roots/herbs are the best remedy which can easily Enlarge your Penis permanently Contact Dr ODABA via Email : Drodabasolutionhome@gmail. com or via WhatsApp : +2347063915230 for Natural root and herbal remedies put together to help you get Enlarge and Erect healthy. Thank you.

      Delete
    3. I will start by saying to all that have COPD or Emphysema should please stop here and read up my story, So as you will know how to get your CURE. I had Chronic Obstructive Pulmonary Disease (COPD) for 15 years, My first symptoms were dry cough, chest tightness and shortness of breath. My first chest x-ray only showed bronchitis. Finally I went to a pulmonologist and was diagnosed with COPD, Am writing this article to appreciate the good work of General Herbal Center, that helped me get read of myCOPD/Emphysema that i have had for about 17years now, with no cure. After seeing a post of a Zom Sanchez from US on the Internet testifying of how she was cured by Native Herbal Clinic prescribed remedies . I also decided to contact then for a cure and i purchased the medicine , because all i wanted was for me to get totally cure and to be free of it all my life. i Am happy today that they helped me and the remedies worked and i can proudly say that i am free from COPD and i am cured completely
      its not worst thing to have COPD/EMPHYSEMA as a begin,you can reach Native Herbal Clinic:(nativehealthclinic@gmail.com or whatsapp +2348140073956

      Delete
  2. OMG this is why i never read movie reviews for movies i'm excited about beforehand; they always prejudice me toward/against the movie!!!

    ReplyDelete
  3. i want to inform the public how i was cured of HERPES Simplex Virus by a Doctor called Dr.EBHOTA. i visited different hospital but they gave me list of drugs like Famvir, Zovirax, and Valtrex which is very expensive to treat the symptoms and never cured me. I was browsing through the Internet searching for remedy on HERPES and i saw comment of people talking about how Doctor EBHOTA cured them. I Was scared because i never believed in the Internet but i was convince to give him a try because i was having no hope of been cured of HERPES so i decided to contact him on his email, i searched his email on net and i saw a lot of people testifying about his goodness. when i contacted him he gave me hope and send a Herbal medicine to me that i took and it seriously worked for me, am a free person now without problem, my HERPES result came out negative. You can contact him on
    his Email: drebhotasolution@gmail.com or whatsapp him on +2348089535482 and get all your problem solved.
    1Sickness of any kind
    2 Help to CURE Hepatitis B, HIV/AIDs CANCER and BIPOLAR
    3 Divorce, Breakup problem and To Re-unit
    4 Pregnancy problem
    5 Financial problem and Job promotion
    6 To get a good and rich life partner
    7 To Lose your wath
    8 to enlargement of pennis
    9. CANCER
    10. ALS (Lou Gehrig's disease)
    11 Hepatitis B
    12 chronic pancreatic
    13 emphysema
    14 COPD (chronic obstructive pulmonary disease)   

    ReplyDelete
  4. i want to inform the public how i was cured of HERPES Simplex Virus by a Doctor called Dr.EBHOTA. i visited different hospital but they gave me list of drugs like Famvir, Zovirax, and Valtrex which is very expensive to treat the symptoms and never cured me. I was browsing through the Internet searching for remedy on HERPES and i saw comment of people talking about how Doctor EBHOTA cured them. I Was scared because i never believed in the Internet but i was convince to give him a try because i was having no hope of been cured of HERPES so i decided to contact him on his email, i searched his email on net and i saw a lot of people testifying about his goodness. when i contacted him he gave me hope and send a Herbal medicine to me that i took and it seriously worked for me, am a free person now without problem, my HERPES result came out negative. You can contact him on
    his Email: drebhotasolution@gmail.com or whatsapp him on +2348089535482 and get all your problem solved.
    1Sickness of any kind
    2 Help to CURE Hepatitis B, HIV/AIDs CANCER and BIPOLAR
    3 Divorce, Breakup problem and To Re-unit
    4 Pregnancy problem
    5 Financial problem and Job promotion
    6 To get a good and rich life partner
    7 To Lose your wath
    8 to enlargement of pennis
    9. CANCER
    10. ALS (Lou Gehrig's disease)
    11 Hepatitis B
    12 chronic pancreatic
    13 emphysema
    14 COPD (chronic obstructive pulmonary disease)   

    ReplyDelete
    Replies
    1. I am very happy today with my family. My name is lisa tommy living in USA, My husband left me for a good 3 years now, and i love him so much, i have been looking for a way to get him back since then. i have tried many options but he did not come back, until i met a friend that darted me to Dr.Jato a spell caster, who helped me to bring back my husband after 2 weeks. Me and my husband are living happily together today, That man is great, you can contact him via email jatolovespell@gmail.com Now i will advice any serious persons that found themselves in this kind of problem to contact him now a fast solution without stress.. He always hello, now i call him my father.contact him now he is always online email (jatolovespell@gmail.com) or contact him on his whatsapp mobile line +2348140033827...

      Delete
  5. I am very happy today with my family. My name is lisa tommy living in USA, My husband left me for a good 3 years now, and i love him so much, i have been looking for a way to get him back since then. i have tried many options but he did not come back, until i met a friend that darted me to Dr.Jato a spell caster, who helped me to bring back my husband after 2 weeks. Me and my husband are living happily together today, That man is great, you can contact him via email jatolovespell@gmail.com Now i will advice any serious persons that found themselves in this kind of problem to contact him now a fast solution without stress.. He always hello, now i call him my father.contact him now he is always online email (jatolovespell@gmail.com) or contact him on his whatsapp mobile line +2348140033827...

    ReplyDelete
  6. Herbal Penis Enlargement product is 100% guarantee to Enlarge and get a better ERECTION ,the reason why most people are finding it difficultto enlarge Penis is because they bedlieve on medicalreport, drugs and medical treatment which is nothelpful for Penis Enlargemaent . Natural roots/herbs are the best remedy which can easily Enlarge your Penis permanently Contact Dr ODABA via Email : Drodabasolutionhome@gmail. com or via WhatsApp : +2347063915230 for Natural root and herbal remedies put together to help you get Enlarge and Erect healthy. Thank you.

    ReplyDelete
  7. I never thought that there will be a miracle on the internet until i came in contact with Priest Babaka, finally i made it with his help, with his spiritual power which was recommended by a lady in baby center he help before, i never believe it was real until i confirm it now because i have tried so many things to make sure i get pregnant but no luck, immediately i contact him. he did some spiritual prayers and send me a medicine which i took, i got pregnant three weeks after, and i now have a son to show. thank you so much priest and i recommend Priest Babaka for everyone out there that is willing to have a child of her own out there. contact him for help too he is real and powerful, i have confirmed it, contact him on email: babaka.wolf@gmail.com or Facebook at priest.babaka





























    I never thought that there will be a miracle on the internet until i came in contact with Priest Babaka, finally i made it with his help, with his spiritual power which was recommended by a lady in baby center he help before, i never believe it was real until i confirm it now because i have tried so many things to make sure i get pregnant but no luck, immediately i contact him. he did some spiritual prayers and send me a medicine which i took, i got pregnant three weeks after, and i now have a son to show. thank you so much priest and i recommend Priest Babaka for everyone out there that is willing to have a child of her own out there. contact him for help too he is real and powerful, i have confirmed it, contact him on email: babaka.wolf@gmail.com or Facebook at priest.babaka

    ReplyDelete
  8. I will start by saying to all that have COPD or Emphysema should please stop here and read up my story, So as you will know how to get your CURE. I had Chronic Obstructive Pulmonary Disease (COPD) for 15 years, My first symptoms were dry cough, chest tightness and shortness of breath. My first chest x-ray only showed bronchitis. Finally I went to a pulmonologist and was diagnosed with COPD, Am writing this article to appreciate the good work of General Herbal Center, that helped me get read of myCOPD/Emphysema that i have had for about 17years now, with no cure. After seeing a post of a Zom Sanchez from US on the Internet testifying of how she was cured by Native Herbal Clinic prescribed remedies . I also decided to contact then for a cure and i purchased the medicine , because all i wanted was for me to get totally cure and to be free of it all my life. i Am happy today that they helped me and the remedies worked and i can proudly say that i am free from COPD and i am cured completely
    its not worst thing to have COPD/EMPHYSEMA as a begin,you can reach Native Herbal Clinic:(nativehealthclinic@gmail.com or whatsapp +2348140073956

    ReplyDelete