Advertisement

COVID-19 randomized controlled trials in medRxiv and PubMed

  • Yuki Kataoka
    Correspondence
    Corresponding author at: Hospital Care Research Unit, Hyogo Prefectural Amagasaki General Medical Center, Higashinaniwa-cho 2-17-77, Amagasaki 660-8550 Japan.
    Affiliations
    Hospital Care Research Unit, Hyogo Prefectural Amagasaki General Medical Center, Higashinaniwa-cho 2-17-77, Amagasaki 660-8550 Japan

    Department of Community Medicine in the Graduate School of Medicine, Kyoto University, Yoshida Konoe-cho, Sakyo-ku, Kyoto 606-8501, Japan

    Systematic Review Workshop Peer Support Group (SRWS-PSG), Japan

    Department of Respiratory Medicine, Hyogo Prefectural Amagasaki General Medical Center, Higashinaniwa-cho 2-17-77, Amagasaki 660-8550 Japan
    Search for articles by this author
  • Shiho Oide
    Affiliations
    Systematic Review Workshop Peer Support Group (SRWS-PSG), Japan

    Department of Gynecology, Women's Center, Yotsuya Medical Cube, 7-7, Nibancho, Chiyoda-ku, Tokyo 102-0084, Japan
    Search for articles by this author
  • Takashi Ariie
    Affiliations
    Systematic Review Workshop Peer Support Group (SRWS-PSG), Japan

    Department of Physical Therapy, School of Health Sciences at Fukuoka, International University of Health and Welfare, 137-1 Enokizu, Okawa-shi, Fukuoka 831-8501, Japan
    Search for articles by this author
  • Yasushi Tsujimoto
    Affiliations
    Systematic Review Workshop Peer Support Group (SRWS-PSG), Japan

    Department of Healthcare Epidemiology, Graduate School of Medicine and Public Health, Kyoto University, Yoshida Konoe-cho, Sakyo-ku, Kyoto 606-8501, Japan

    Department of Nephrology and Dialysis, Kyoritsu Hospital, 16-5 Chuo-cho, Kawanishi, Hyogo 666-0016, Japan
    Search for articles by this author
  • Toshi A. Furukawa
    Affiliations
    Department of Health Promotion and Human Behavior, Kyoto University Graduate School of Medicine / School of Public Health, Yoshida Konoe-cho, Sakyo-ku, Kyoto 606-8501, Japan
    Search for articles by this author
Published:September 22, 2020DOI:https://doi.org/10.1016/j.ejim.2020.09.019

      Highlights

      • This study investigated the difference between COVID-19 RCT in medRxiv and in PubMed.
      • SPIN in the conclusion was more frequently seen in reports in medRxiv than PubMed.
      • Readers should pay attention to the overstatements in preprints of COVID-19 RCT.
      Dear editor,
      Fast-track preprints have garnered increased attention since the COVID-19 pandemic. However, little is known about the quality of clinical research published as preprints. The objective of this study is to explore the differences in COVID-19 randomized controlled trial (RCT) articles between medRxiv and PubMed.
      This is a meta-epidemiological investigation. We published the protocol prior to conducting this study [

      Kataoka Y., Oide S., Ariie T., Tsujimoto Y., Furukawa T.A.. Quality of COVID-19 research in preprints: a meta-epidemiological study protocol. protocols.io.2020. doi:10.17504/protocols.io.bhm8j49w.

      ]. S1 Appendix is the reporting checklist [
      • Murad M.H.
      • Wang Z.
      Guidelines for reporting meta-epidemiological methodology research.
      ]. We included RCT articles covering topics related to the COVID-19 practice indexed in PubMed or MedRxiv from 1st January 2020 to 15th June. We searched abstracts that used the word “random” from medRxiv COVID-19 SARS-CoV-2 preprints [

      medRxiv COVID-19 SARS-CoV-2 preprints from medRxiv and bioRxiv.https://connect.medrxiv.org/relate/content/181. Accessed July 3, 2020.

      ]. We searched abstracts from PubMed using Shokraneh's filter for COVID-19 [
      • Shokraneh F.
      Keeping up with studies on covid-19: systematic search strategies and resources.
      ] and Cochrane filter for identifying randomized trials [

      PubMed | Cochrane Work. https://work.cochrane.org/pubmed. Accessed June 3, 2020.

      ]. Two of three review authors (YK, SO, TA) selected abstracts from search results independently. Disagreements were resolved through discussion. One of three review authors (YK, SO, TA) selected full text articles and extracted data after calibration exercises. Disagreements were resolved through discussion. We evaluated the characteristics of included articles and the risk of bias using risk of bias tool 2 for the registered first primary outcome, or the outcome presented first in reports without the registration information [
      • Sterne J.A.C.
      • Savović J.
      • Page M.J.
      • et al.
      RoB 2: a revised tool for assessing risk of bias in randomised trials.
      ]. We judged the presence of SPIN when there were positive expressions in the title or abstract conclusion in studies whose pre-registered primary outcome was non-significant [
      • Boutron I.
      Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes.
      ]. We estimated the risk differences and confidence intervals for binary variables. We used Wilcoxon rank sum test for continuous variables. A p-value < .05 was considered statistically significant.
      Fig. 1 shows the study selection procedure. We included a total of 29 RCT articles (13 from medRxiv, and 16 from PubMed). Table 1 shows the summary characteristics of the included articles. The citations of included articles and details of evaluations are shown in S2 Appendix. Three articles from PubMed did not show the trial registration numbers. Overall, more than 70% (21/29) articles were at high or some concerns for risk of bias, especially in the domain of selection of the reported results. There were 11 articles whose pre-registered primary outcomes were negative. Among them, there were four articles with SPIN (80%) indexed in medRxiv and another (17%) indexed in PubMed. For example, one article in medRxiv stated “… effects in COVID-19 may be clinically important and warrant further consideration and studies” based on the result of the primary outcome that was not statistically significant (odds ratio 0.59, 95% CI 0.148 to 2.352 P = .454) [
      • Ye Y.
      Guideline-based Chinese herbal medicine treatment plus standard care for severe coronavirus disease 2019 (G-CHAMPS): evidence from China.
      ]. There was no statistically significant difference between the number of mentions on social networking sites (median [interquartile range (IQR)] in medRvix 80 [5-245] vs. PubMed 342.5 [78.5-5197], p = .09) or citations (median [IQR] in medRvix 0 [0-39] vs. PubMed 10.5 [2.5-47], p = .17) between the two sources. Two pairs of articles were published both in medRxiv and peer-reviewed journal. One of them discussed limitations in the abstract in PubMed but not in medRxiv, apparently after peer-review.
      Table 1Characteristics of included articles.
      medRxivPubMedTotalRisk differences (%) and confidence intervalsg
      N = 13N = 16N = 29
      Number of participants of full analysis set81 (42–92)115 (54.5–260.5)86 (50–199).24h
      Without the information of trial registration0 (0%)3 (19%)3 (10%)19 [−0.37 to 38]
      Inconsistent with trial registrationa8 (62%)4 (30%)12 (46%)−31 [−67 to 5.7]
      Referring limitations in abstracts
       Present1 (8%)4 (25%)5 (17%)19 [−7.7 to 46]
       Without an abstract0 (0%)1 (6%)1 (3%)
      High or some concerns of risk of biasb
       Randomization process3 (23%)5 (31%)8 (28%)8.2 [−24 to 40]
       Deviations from intended interventions3 (23%)4 (25%)7 (24%)1.9 [−29 to 33]
       Missing outcome data2 (15%)0 (0%)2 (7%)−15 [−35 to 4.2]
       Measurement of the outcome6 (46%)6 (38%)13 (45%)−8.7 [−45 to 27]
       Selection of the reported result9 (69%)8 (50%)17 (60%)−19 [−54 to 16]
       Overall Bias10 (77%)11 (70%)21 (72%)−8.2 [−40 to 24]
      SPINc in titles (n = 11)d
       Present1 (20%)0 (0%)1 (3%)−20 [−55 to 15]
      SPINc in conclusions (n = 11) d
       present4 (80%)1 (17%)5 (45%)−63 [−100 to −17]
      Number of SNS sharee.09h
       03 (23%)2 (13%)5 (17%)
       10> ≥11 (8%)1 (6%)2 (7%)
       100> ≥103 (23%)1 (6%)2 (14%)
       1000> ≥1005 (38%)5 (31%)10 (34%)
       10,000> ≥10001 (8%)4 (25%)5 (17%)
       ≥10,0000 (0%)3 (19%)3 (10%)
      Number of citationsf.17h
       07 (54%)3 (19%)10 (34%)
       10> ≥11 (8%)5 (31%)6 (21%)
       100> ≥104 (31%)5 (31%)9 (31%)
       ≥1001 (8%)3 (19%)4 (14%)

      1. Discussion

      Our results suggest the existence of RCT articles that had problems with research methods and reporting in both medRxiv and PubMed journals related to the COVID-19 practice. In particular, the SPIN in the abstract conclusion was more frequently seen in reports in medRxiv.
      This study has several imitations, including small sample size, and not including preprint servers other than medRxiv.
      Further investigation on the impact of accelerated publication without peer review on the quality of reporting of clinical studies is warranted. Readers should pay attention to the overstatements in preprints of RCT. mmc1.docx mmc2.xlsx

      Compliance with ethical stantards

      Because all data were retrieved from public databases, institutional review board approval is not required.

      Author contributions

      Yuki Kataoka had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
      Concept and design: All authors.
      Acquisition, analysis, or interpretation of data: All authors.
      Drafting of the manuscript: Yuki Kataoka.
      Critical revision of the manuscript for important intellectual content: All authors.
      Supervision: Toshi A. Furukawa.

      Funding

      Self funding.

      Declaration of Competing Interest

      Toshi A. Furukawa reports personal fees from Meiji, Mitsubishi-Tanabe, MSD and Pfizer and a grant from Mitsubishi-Tanabe, outside the submitted work; TAF has a patent 2018-177688 pending. Other authors declare that they have no conflict of interest.

      Acknowledgement

      We acknowledge Ms. Kyoko Wasai, who assisted retrieving full text articles.

      Appendix. Supplementary materials

      References

      1. Kataoka Y., Oide S., Ariie T., Tsujimoto Y., Furukawa T.A.. Quality of COVID-19 research in preprints: a meta-epidemiological study protocol. protocols.io.2020. doi:10.17504/protocols.io.bhm8j49w.

        • Murad M.H.
        • Wang Z.
        Guidelines for reporting meta-epidemiological methodology research.
        Evid Based Med. 2017; 22: 139-142https://doi.org/10.1136/ebmed-2017-110713
      2. medRxiv COVID-19 SARS-CoV-2 preprints from medRxiv and bioRxiv.https://connect.medrxiv.org/relate/content/181. Accessed July 3, 2020.

        • Shokraneh F.
        Keeping up with studies on covid-19: systematic search strategies and resources.
        BMJ. 2020; (Aprilm1601)https://doi.org/10.1136/bmj.m1601
      3. PubMed | Cochrane Work. https://work.cochrane.org/pubmed. Accessed June 3, 2020.

        • Sterne J.A.C.
        • Savović J.
        • Page M.J.
        • et al.
        RoB 2: a revised tool for assessing risk of bias in randomised trials.
        BMJ. 2019; (August): l4898https://doi.org/10.1136/bmj.l4898
        • Boutron I.
        Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes.
        JAMA. 2010; 303: 2058https://doi.org/10.1001/jama.2010.651
        • Ye Y.
        Guideline-based Chinese herbal medicine treatment plus standard care for severe coronavirus disease 2019 (G-CHAMPS): evidence from China.
        medRxiv. 2020; (January 202003.27.20044974)https://doi.org/10.1101/2020.03.27.20044974