Advertisement

Active smoking and severity of coronavirus disease 2019 (COVID-19): The use of significance testing leads to an erroneous conclusion

  • Ernest Lo
    Correspondence
    Corresponding author.
    Affiliations
    Institut National de Santé Publique du Québec, 190 Boulevard Crémazie Est, Montréal, Québec, H2P 1E2, Canada

    Department of Epidemiology, Biostatistics and Occupational Health, McGill University, Purvis Hall, 1020 Pine Avenue West, Montréal, Québec, H3A 1A2, Canada
    Search for articles by this author
  • Benoit Lasnier
    Affiliations
    Institut National de Santé Publique du Québec, 190 Boulevard Crémazie Est, Montréal, Québec, H2P 1E2, Canada
    Search for articles by this author
      Dear Editor,
      In their letter to the Editor, “Active smoking is not associated with severity of coronavirus disease 2019 (COVID-19)”, Lippi et al. [
      • Lippi G
      • Henry BM.
      Active smoking is not associated with severity of coronavirus disease 2019 (COVID-19).
      ] present a timely and important meta-analysis to evaluate if smokers are at greater risk of increased severity of COVID-19 symptoms. This is the first study to assess this question and has important implications for clinical practice and public health recommendations. It is indeed imperative at this time to identify if and how smoking habits can contribute to the progression of COVID-19 and its ensuing complications, such as hospitalisation, admission to an intensive care unit, and ultimately death.
      Unfortunately, in their statistical analysis, Lippi et al. use a null hypothesis significance testing (NHST) approach which is known to have fundamental flaws as discussed in recent high profile statements [
      • Wasserstein RL
      • Schirm AL
      • Lazar NA
      Moving to a world beyond “p < 0.05”.
      ,
      • Wasserstein RL
      • Lazar NA.
      The ASA’s statement on p-values: context, process, and purpose.
      ]. The principal use of NHST leads them to erroneously conclude an absence of effect of smoking, which is incorrect, and could have potentially harmful consequences. Lippi et al. estimate a pooled OR (95% confidence interval) of 1.69 (0.41, 6.92) for the risk of increased severity of COVID-19 symptoms due to smoking. Since the lower limit of the OR overlaps the null value of 1, Lippi et al. are led to conclude that “active smoking is not associated with severity of coronavirus disease 2019” (cf. the title of their letter). While the presentation of confidence intervals is commendable, the comparison of a confidence interval limit with a null value to determine whether or not an effect exists is essentially a surrogate for a null hypothesis test with level α=5% [
      • Fletcher RH
      • Fletcher SW
      • Fletcher GS
      Clinical Epidemiology: the Essentials.
      ]. This practice has been strongly criticized and described as a “misuse” or “debasing” of the confidence interval [
      • Rothman KJ.
      Disengaging from statistical significance.
      ].
      The estimation approach has been proposed as a more correct and clinically informative statistical approach for interpreting the results of medical studies [
      • Gardner MJ
      • Altman DG
      Confidence intervals rather than P values: estimation rather than hypothesis testing.
      ,
      • Cumming G.
      The new statistics: why and how.
      ]. In contrast to NHST, this approach correctly recognizes that the entire confidence interval represents a range of plausible values for the OR [
      • Gardner MJ
      • Altman DG
      Confidence intervals rather than P values: estimation rather than hypothesis testing.
      ,
      • Cumming G.
      The new statistics: why and how.
      ]. Thus while the point estimate of 1.69 of the Lippi et al. study represents the expected or mean value, the OR has a probability distribution that extends to either side such that values from 0.41 to 6.92 are also quite probable [
      • Cumming G.
      The new statistics: why and how.
      ]. In fact the bulk of the confidence interval extends well beyond the null value of 1 (Fig. 1). Mathematical integration of the OR probability distribution (assuming a Normal distribution for the log odds ratio) indicates a 77% probability that the estimated OR exceeds 1, while the probability of it exceeding hypothetical clinical thresholds of 1.2 and 2.0 are 68% and 41% respectively, as shown in Fig. 1. These results indicate there is a substantial probability of a clinically important effect of smoking on COVID-19 severity, in contrast with the conclusions presented by Lippi et al.
      The application of the estimation approach to the OR estimated in their sensitivity analysis leads to an even more striking difference in interpretation. Exclusion of the largest study of the five studies results in a pooled OR of 4.35 (0.86, 21.86). While Lippi et al. again perform a null hypothesis test and conclude that there is no association, almost the entire range of plausible values, as indicated by the confidence interval, lies well above the null OR value of 1 (Fig. 2). It can be calculated that 96% of the OR probability distribution rests above the null, and that the probability of exceeding hypothetical clinical thresholds of 1.2 and 2.0 are 94% and 83% respectively.
      The inappropriateness of using null hypothesis significance testing to conclude an absence of effect has been well documented in the clinical literature [
      • Altman DG
      • Bland JM.
      Absence of evidence is not evidence of absence.
      ,
      • Freiman JA
      • Chalmers TC
      • Smith H
      • Kuebler RR
      The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial: survey of 71 negative trials.
      ]. The Lippi et al. study is analogous to an underpowered clinical trial where elevated variability, as reflected in the wide confidence intervals, precludes the ruling out of a non-null or clinically important association. A more appropriate conclusion for this study is that a clinically important association is uncertain due to the lack of sample size, but that the effect of smoking on COVID-19 severity remains highly possible; further and larger studies should be encouraged. The clinical and practical message of this conclusion is substantially different from the negative result asserted in both the conclusion and title of the Lippi et al. letter.
      It would be accurate for Lippi et al. to only state that there is no statistically significant association (at the 5% level) based on their meta-analysis, as this is simply a statistical result that makes no claim as to what effect sizes may be plausible. However Lippi et al. also use the phrase “no significant association” in their text, omitting the ‘statistical’ qualifier and leading readers to suppose erroneously that the magnitude of association is clinically unimportant. This distortion of meaning is exacerbated in the title of their letter which emphatically states that “active smoking is not associated with severity of coronavirus disease 2019 (COVID-19)”. The faulty translation of NHST results into statements with clinical significance can mislead readers and lead to biases in future research [
      • Wasserstein RL
      • Schirm AL
      • Lazar NA
      Moving to a world beyond “p < 0.05”.
      ]. Indeed two subsequent meta-analyses on the effect of smoking on COVID-19 have already cited the Lippi et al. study as an example of a negative result [
      • Patanavanich R
      • Glantz SA.
      Smoking is associated with COVID-19 progression: a meta-analysis.
      ,
      • Zhao Q
      • Meng M
      • Kumar R
      • Wu Y
      • Huang J
      • Lian N
      • et al.
      The impact of COPD and smoking history on the severity of Covid-19: A systemic review and meta-analysis.
      ].
      Amidst the pressing need to conduct and disseminate new research on COVID-19, there remains more than ever the need for researchers and practitioners of evidence based medicine to apply sound methods of statistical inference in the interpretation of their results. Null hypothesis significance testing, without consideration of the range of plausible effect sizes, is known to produce conclusions that are non-reproducible and represent over-simplistic dichotomies that can impede scientific progress [
      • Wasserstein RL
      • Schirm AL
      • Lazar NA
      Moving to a world beyond “p < 0.05”.
      ,
      • Wasserstein RL
      • Lazar NA.
      The ASA’s statement on p-values: context, process, and purpose.
      ]. The estimation approach with the proper use of confidence intervals is greatly preferred, and in this case, indicates there is a substantial possibility of an association between smoking and COVID-19 severity.

      Declaration of Competing Interest

      Re: Active smoking and severity of coronavirus disease (COVID-19): the use of significance testing leads to an erroneous conclusion
      The authors Ernest Lo and Benoit Lasnier declare that they have no conflict of interest.

      Acknowledgements

      The authors would like to thank Denis Hamel and Shu Qin Wei for their helpful comments on a previous version of this manuscript.

      References

        • Lippi G
        • Henry BM.
        Active smoking is not associated with severity of coronavirus disease 2019 (COVID-19).
        Eur J Intern Med. 2020; 75: 107-108https://doi.org/10.1016/j.ejim.2020.03.014
        • Wasserstein RL
        • Schirm AL
        • Lazar NA
        Moving to a world beyond “p < 0.05”.
        Am Stat. 2019; 73: 1-19https://doi.org/10.1080/00031305.2019.1583913
        • Wasserstein RL
        • Lazar NA.
        The ASA’s statement on p-values: context, process, and purpose.
        Am Stat. 2016; 70: 129-133https://doi.org/10.1080/00031305.2016.1154108
        • Fletcher RH
        • Fletcher SW
        • Fletcher GS
        Clinical Epidemiology: the Essentials.
        5th ed. Philadelphia: Wolters Kluwer/Lippincott Williams & Wilkins Health, 2014
        • Rothman KJ.
        Disengaging from statistical significance.
        Eur J Epidemiol. 2016; 31: 443-444https://doi.org/10.1007/s10654-016-0158-2
        • Gardner MJ
        • Altman DG
        Confidence intervals rather than P values: estimation rather than hypothesis testing.
        Br Med J (Clin Res Ed). 1986; 292: 746-750
        • Cumming G.
        The new statistics: why and how.
        Psychol Sci. 2014; 25: 7-29https://doi.org/10.1177/0956797613504966
        • Altman DG
        • Bland JM.
        Absence of evidence is not evidence of absence.
        BMJ. 1995; 311: 485
        • Freiman JA
        • Chalmers TC
        • Smith H
        • Kuebler RR
        The importance of beta, the type II error and sample size in the design and interpretation of the randomized control trial: survey of 71 negative trials.
        N Engl J Med. 1978; 299: 690-694https://doi.org/10.1056/NEJM197809282991304
        • Patanavanich R
        • Glantz SA.
        Smoking is associated with COVID-19 progression: a meta-analysis.
        Public Glob. Health. 2020; https://doi.org/10.1101/2020.04.13.20063669
        • Zhao Q
        • Meng M
        • Kumar R
        • Wu Y
        • Huang J
        • Lian N
        • et al.
        The impact of COPD and smoking history on the severity of Covid-19: A systemic review and meta-analysis.
        J Med Virol. 2020; https://doi.org/10.1002/jmv.25889