If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
Corresponding author at: Hospital Care Research Unit, Hyogo Prefectural Amagasaki General Medical Center, Higashinaniwa-cho 2-17-77, Amagasaki 660-8550 Japan.
Hospital Care Research Unit, Hyogo Prefectural Amagasaki General Medical Center, Higashinaniwa-cho 2-17-77, Amagasaki 660-8550 JapanDepartment of Community Medicine in the Graduate School of Medicine, Kyoto University, Yoshida Konoe-cho, Sakyo-ku, Kyoto 606-8501, JapanSystematic Review Workshop Peer Support Group (SRWS-PSG), JapanDepartment of Respiratory Medicine, Hyogo Prefectural Amagasaki General Medical Center, Higashinaniwa-cho 2-17-77, Amagasaki 660-8550 Japan
Systematic Review Workshop Peer Support Group (SRWS-PSG), JapanDepartment of Gynecology, Women's Center, Yotsuya Medical Cube, 7-7, Nibancho, Chiyoda-ku, Tokyo 102-0084, Japan
Systematic Review Workshop Peer Support Group (SRWS-PSG), JapanDepartment of Physical Therapy, School of Health Sciences at Fukuoka, International University of Health and Welfare, 137-1 Enokizu, Okawa-shi, Fukuoka 831-8501, Japan
Systematic Review Workshop Peer Support Group (SRWS-PSG), JapanDepartment of Healthcare Epidemiology, Graduate School of Medicine and Public Health, Kyoto University, Yoshida Konoe-cho, Sakyo-ku, Kyoto 606-8501, JapanDepartment of Nephrology and Dialysis, Kyoritsu Hospital, 16-5 Chuo-cho, Kawanishi, Hyogo 666-0016, Japan
Department of Health Promotion and Human Behavior, Kyoto University Graduate School of Medicine / School of Public Health, Yoshida Konoe-cho, Sakyo-ku, Kyoto 606-8501, Japan
This study investigated the difference between COVID-19 RCT in medRxiv and in PubMed.
•
SPIN in the conclusion was more frequently seen in reports in medRxiv than PubMed.
•
Readers should pay attention to the overstatements in preprints of COVID-19 RCT.
Dear editor,
Fast-track preprints have garnered increased attention since the COVID-19 pandemic. However, little is known about the quality of clinical research published as preprints. The objective of this study is to explore the differences in COVID-19 randomized controlled trial (RCT) articles between medRxiv and PubMed.
This is a meta-epidemiological investigation. We published the protocol prior to conducting this study [
Kataoka Y., Oide S., Ariie T., Tsujimoto Y., Furukawa T.A.. Quality of COVID-19 research in preprints: a meta-epidemiological study protocol. protocols.io.2020. doi:10.17504/protocols.io.bhm8j49w.
]. We included RCT articles covering topics related to the COVID-19 practice indexed in PubMed or MedRxiv from 1st January 2020 to 15th June. We searched abstracts that used the word “random” from medRxiv COVID-19 SARS-CoV-2 preprints [
]. Two of three review authors (YK, SO, TA) selected abstracts from search results independently. Disagreements were resolved through discussion. One of three review authors (YK, SO, TA) selected full text articles and extracted data after calibration exercises. Disagreements were resolved through discussion. We evaluated the characteristics of included articles and the risk of bias using risk of bias tool 2 for the registered first primary outcome, or the outcome presented first in reports without the registration information [
]. We judged the presence of SPIN when there were positive expressions in the title or abstract conclusion in studies whose pre-registered primary outcome was non-significant [
]. We estimated the risk differences and confidence intervals for binary variables. We used Wilcoxon rank sum test for continuous variables. A p-value < .05 was considered statistically significant.
Fig. 1 shows the study selection procedure. We included a total of 29 RCT articles (13 from medRxiv, and 16 from PubMed). Table 1 shows the summary characteristics of the included articles. The citations of included articles and details of evaluations are shown in S2 Appendix. Three articles from PubMed did not show the trial registration numbers. Overall, more than 70% (21/29) articles were at high or some concerns for risk of bias, especially in the domain of selection of the reported results. There were 11 articles whose pre-registered primary outcomes were negative. Among them, there were four articles with SPIN (80%) indexed in medRxiv and another (17%) indexed in PubMed. For example, one article in medRxiv stated “… effects in COVID-19 may be clinically important and warrant further consideration and studies” based on the result of the primary outcome that was not statistically significant (odds ratio 0.59, 95% CI 0.148 to 2.352 P = .454) [
]. There was no statistically significant difference between the number of mentions on social networking sites (median [interquartile range (IQR)] in medRvix 80 [5-245] vs. PubMed 342.5 [78.5-5197], p = .09) or citations (median [IQR] in medRvix 0 [0-39] vs. PubMed 10.5 [2.5-47], p = .17) between the two sources. Two pairs of articles were published both in medRxiv and peer-reviewed journal. One of them discussed limitations in the abstract in PubMed but not in medRxiv, apparently after peer-review.
Our results suggest the existence of RCT articles that had problems with research methods and reporting in both medRxiv and PubMed journals related to the COVID-19 practice. In particular, the SPIN in the abstract conclusion was more frequently seen in reports in medRxiv.
This study has several imitations, including small sample size, and not including preprint servers other than medRxiv.
Further investigation on the impact of accelerated publication without peer review on the quality of reporting of clinical studies is warranted. Readers should pay attention to the overstatements in preprints of RCT. mmc1.docx mmc2.xlsx
Compliance with ethical stantards
Because all data were retrieved from public databases, institutional review board approval is not required.
Author contributions
Yuki Kataoka had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: All authors.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Yuki Kataoka.
Critical revision of the manuscript for important intellectual content: All authors.
Supervision: Toshi A. Furukawa.
Funding
Self funding.
Declaration of Competing Interest
Toshi A. Furukawa reports personal fees from Meiji, Mitsubishi-Tanabe, MSD and Pfizer and a grant from Mitsubishi-Tanabe, outside the submitted work; TAF has a patent 2018-177688 pending. Other authors declare that they have no conflict of interest.
Acknowledgement
We acknowledge Ms. Kyoko Wasai, who assisted retrieving full text articles.
Kataoka Y., Oide S., Ariie T., Tsujimoto Y., Furukawa T.A.. Quality of COVID-19 research in preprints: a meta-epidemiological study protocol. protocols.io.2020. doi:10.17504/protocols.io.bhm8j49w.
30371-X&id=gr1.jpg){kind=link}