Advertisement

Venous lactate improves the prediction of in-hospital adverse outcomes in normotensive pulmonary embolism

  • Matthias Ebner
    Affiliations
    Department of Cardiology and Angiology, Campus Charité Mitte (CCM), Charité – University Medicine Berlin, Germany

    German Center for Cardiovascular Research (DZHK), Partner site Berlin, Germany
    Search for articles by this author
  • Charlotta F. Pagel
    Affiliations
    Clinic of Cardiology and Pneumology, University Medical Center Göttingen, Germany
    Search for articles by this author
  • Carmen Sentler
    Affiliations
    Clinic of Cardiology and Pneumology, University Medical Center Göttingen, Germany
    Search for articles by this author
  • Veli-Pekka Harjola
    Affiliations
    University of Helsinki, Emergency Medicine, Department of Emergency Medicine and Services, Helsinki University Hospital, Helsinki, Finland
    Search for articles by this author
  • Héctor Bueno
    Affiliations
    Centro Nacional de Investigaciones Cardiovasculares (CNIC), Madrid, Spain

    Cardiology Department, Hospital Universitario 12 de Octubre and Instituto de Investigación Sanitaria Hospital 12 de Octubre (imas12), Madrid, Spain

    Facultad de Medicina, Universidad Complutense de Madrid, Madrid, Spain
    Search for articles by this author
  • Markus H. Lerchbaumer
    Affiliations
    Department of Radiology, Campus Charité Mitte (CCM), Charité – University Medicine Berlin, Germany
    Search for articles by this author
  • Karl Stangl
    Affiliations
    Department of Cardiology and Angiology, Campus Charité Mitte (CCM), Charité – University Medicine Berlin, Germany
    Search for articles by this author
  • Burkert Pieske
    Affiliations
    German Center for Cardiovascular Research (DZHK), Partner site Berlin, Germany

    Department of Internal Medicine and Cardiology, Campus Virchow Klinikum (CVK), Charité – University Medicine Berlin, Germany

    Berlin Institute of Health, Berlin, Germany
    Search for articles by this author
  • Gerd Hasenfuß
    Affiliations
    Clinic of Cardiology and Pneumology, University Medical Center Göttingen, Germany

    German Center for Cardiovascular Research (DZHK), Partner site Göttingen, Germany
    Search for articles by this author
  • Stavros V. Konstantinides
    Affiliations
    Center for Thrombosis and Hemostasis (CTH), University Medical Center Mainz, Germany

    Department of Cardiology, Democritus University of Thrace, Alexandroupolis, Greece
    Search for articles by this author
  • Mareike Lankeit
    Correspondence
    Corresponding author: Department of Internal Medicine and Cardiology, Campus Virchow Klinikum (CVK), Charité – University Medicine Berlin, Augustenburger Platz 1, 13353 Berlin, Germany.
    Affiliations
    German Center for Cardiovascular Research (DZHK), Partner site Berlin, Germany

    Clinic of Cardiology and Pneumology, University Medical Center Göttingen, Germany

    Department of Internal Medicine and Cardiology, Campus Virchow Klinikum (CVK), Charité – University Medicine Berlin, Germany

    Center for Thrombosis and Hemostasis (CTH), University Medical Center Mainz, Germany
    Search for articles by this author
Open AccessPublished:February 05, 2021DOI:https://doi.org/10.1016/j.ejim.2021.01.021

      Highlights

      • Elevated venous lactate predicts adverse outcomes in normotensive pulmonary embolism.
      • Optimal prognostic performance was observed for venous lactate ≥3.3 mmol/l.
      • Adding venous lactate to the 2019 ESC algorithm may improve risk stratification.
      • The established threshold for arterial lactate has limited specificity in venous samples.

      Abstract

      Background

      Arterial lactate is an established risk marker in patients with pulmonary embolism (PE). However, its clinical applicability is limited by the need of an arterial puncture. In contrast, venous lactate can easily be measured from blood samples obtained via routine peripheral venepuncture.

      Methods

      We investigated the prognostic value of venous lactate with regard to in-hospital adverse outcomes and mortality in 419 consecutive PE patients enrolled in a single-center registry between 09/2008 and 09/2017.

      Results

      An optimised venous lactate cut-off value of 3.3 mmol/l predicted both, in-hospital adverse outcome (OR 11.0 [95% CI 4.6–26.3]) and all-cause mortality (OR 3.8 [95%CI 1.3–11.3]). The established cut-off value for arterial lactate (2.0 mmol/l) and the upper limit of normal for venous lactate (2.3 mmol/l) had lower prognostic value for adverse outcomes (OR 3.6 [95% CI 1.5–8.7] and 5.7 [95% CI 2.4–13.6], respectively) and did not predict mortality. If added to the 2019 European Society of Cardiology (ESC) algorithm, venous lactate <2.3 mmol/l was associated with a high negative predictive value (0.99 [95% CI 0.97–1.00]) for adverse outcomes in intermediate-low-risk patients, whereas levels ≥3.3 mmol/l predicted adverse outcomes in the intermediate-high-risk group (OR 5.2 [95% CI 1.8–15.0]).

      Conclusion

      Venous lactate above the upper limit of normal was associated with increased risk for adverse outcomes and an optimised cut-off value of 3.3 mmol/l predicted adverse outcome and mortality. Adding venous lactate to the 2019 ESC algorithm may improve risk stratification. Importantly, the established cut-off value for arterial lactate has limited specificity in venous samples and should not be used.

      Keywords

      1. Introduction

      Pulmonary embolism (PE) is associated with high morbidity and mortality, making it a major contributor to global disease burden. [
      • Raskob G.E.
      • Angchaisuksiri P.
      • Blanco A.N.
      • Buller H.
      • Gallus A.
      • Hunt B.J.
      • et al.
      Thrombosis: a major contributor to global disease burden.
      ,
      • Konstantinides S.V.
      • Meyer G.
      • Becattini C.
      • Bueno H.
      • Geersing G.J.
      • Harjola V.P.
      • et al.
      ESC Guidelines for the diagnosis and management of acute pulmonary embolism developed in collaboration with the European Respiratory Society (ERS).
      ] Thus, current guideline recommendations emphasize the importance of early risk stratification in the heterogeneous group of normotensive PE patients to guide therapeutic decision making. [
      • Konstantinides S.V.
      • Meyer G.
      • Becattini C.
      • Bueno H.
      • Geersing G.J.
      • Harjola V.P.
      • et al.
      ESC Guidelines for the diagnosis and management of acute pulmonary embolism developed in collaboration with the European Respiratory Society (ERS).
      ] The main determinant of adverse outcomes in PE patients is right ventricular (RV) failure due to the occlusion of the pulmonary vasculature by embolised thrombi, that may progress to manifest obstructive shock. [
      • Harjola V.P.
      • Mebazaa A.
      • Celutkiene J.
      • Bettex D.
      • Bueno H.
      • Chioncel O.
      • et al.
      Contemporary management of acute right ventricular failure: a statement from the Heart Failure Association and the Working Group on Pulmonary Circulation and Right Ventricular Function of the European Society of Cardiology.
      ] Lactate is a marker of the adequacy of tissue perfusion and has been shown to correlate with disease severity in a variety of shock states, ranging from sepsis to trauma and cardiogenic shock. [
      • Thiele H.
      • Ohman E.M.
      • Desch S.
      • Eitel I.
      • de Waha S.
      Management of cardiogenic shock.
      ,
      • Regnier M.A.
      • Raux M.
      • Le Manach Y.
      • Asencio Y.
      • Gaillard J.
      • Devilliers C.
      • et al.
      Prognostic significance of blood lactate and lactate clearance in trauma patients.
      ,
      • Casserly B.
      • Phillips G.S.
      • Schorr C.
      • Dellinger R.P.
      • Townsend S.R.
      • Osborn T.M.
      • et al.
      Lactate measurements in sepsis-induced tissue hypoperfusion: results from the Surviving Sepsis Campaign database.
      ] Previous reports indicate that arterial lactate levels at presentation predict outcome in acute PE patients and can be used to improve existing risk stratification algorithms. [
      • Vanni S.
      • Socci F.
      • Pepe G.
      • Nazerian P.
      • Viviani G.
      • Baioni M.
      • et al.
      High plasma lactate levels are associated with increased risk of in-hospital mortality in patients with pulmonary embolism.
      ,
      • Vanni S.
      • Viviani G.
      • Baioni M.
      • Pepe G.
      • Nazerian P.
      • Socci F.
      • et al.
      Prognostic value of plasma lactate levels among patients with acute pulmonary embolism: the thrombo-embolism lactate outcome study.
      ,
      • Vanni S.
      • Jimenez D.
      • Nazerian P.
      • Morello F.
      • Parisi M.
      • Daghini E.
      • et al.
      Short-term clinical outcome of normotensive patients with acute PE and high plasma lactate.
      ,
      • Vanni S.
      • Nazerian P.
      • Bova C.
      • Bondi E.
      • Morello F.
      • Pepe G.
      • et al.
      Comparison of clinical scores for identification of patients with pulmonary embolism at intermediate-high risk of adverse clinical outcome: the prognostic role of plasma lactate.
      ] Although these results appear promising, their clinical application is restricted by the need to obtain arterial blood samples. While peripheral venous access is routinely established in all emergency patients, the need of an arterial puncture demands additional time, can be technically challenging and, importantly, increases the risk of bleeding if thrombolysis is performed due to secondary hemodynamic instability.
      Although much easier to obtain, lactate concentrations in peripheral venous samples may differ from arterial levels, especially in settings of inadequate tissue perfusion. [
      • Kruse O.
      • Grunnet N.
      • Barfod C.
      Blood lactate as a predictor for in-hospital mortality in patients admitted acutely to hospital: a systematic review.
      ] Prior investigations comparing arterial and venous lactate levels in emergency patients reported a tendency of venous lactate to be higher than arterial lactate. [
      • Younger J.G.
      • Falk J.L.
      • Rothrock S.G.
      Relationship between arterial and peripheral venous lactate levels.
      ,
      • Mikami A.
      • Ohde S.
      • Deshpande G.A.
      • Mochizuki T.
      • Otani N.
      • Ishimatsu S.
      Can we predict arterial lactate from venous lactate in the ED?.
      ] Hence, concerns arose that using the same cut-off values for venous and arterial lactate may lead to false positive results. [
      • Kruse O.
      • Grunnet N.
      • Barfod C.
      Blood lactate as a predictor for in-hospital mortality in patients admitted acutely to hospital: a systematic review.
      ] Therefore, it remains uncertain whether the prognostic value of arterial lactate also applies to venous lactate and if the same cut-off values can be used to risk-stratify patients with acute PE. Thus, we aimed to investigate the prognostic value of peripheral venous lactate concentrations for the prediction of adverse outcomes in acute PE, to establish an optimised cut-off value and evaluate the potential benefits of venous lactate in addition to existing risk stratification algorithms.

      2. Materials and methods

      2.1 Study design and definition of outcomes

      The Pulmonary Embolism Registry of Göttingen (PERGO) prospectively includes consecutive patients with objectively confirmed PE ≥18 years of age admitted to the University Medical Center Göttingen, Germany. The study protocol has been described in detail previously. [
      • Hellenkamp K.
      • Pruszczyk P.
      • Jimenez D.
      • Wyzgal A.
      • Barrios D.
      • Ciurzynski M.
      • et al.
      Prognostic impact of copeptin in pulmonary embolism: a multicentre validation study.
      ,
      • Ebner M.
      • Kresoja K.P.
      • Keller K.
      • Hobohm L.
      • Rogge N.I.J.
      • Hasenfuss G.
      • et al.
      Temporal trends in management and outcome of pulmonary embolism: a single-centre experience.
      ] The present analysis included patients enrolled in PERGO between September 2008 and March 2018. We excluded patients with (i) missing peripheral venous lactate measurements within six hours of presentation, (ii) high-risk PE according to the definition provided in the 2019 ESC guidelines [
      • Konstantinides S.V.
      • Meyer G.
      • Becattini C.
      • Bueno H.
      • Geersing G.J.
      • Harjola V.P.
      • et al.
      ESC Guidelines for the diagnosis and management of acute pulmonary embolism developed in collaboration with the European Respiratory Society (ERS).
      ], (iii) significant concomitant acute cardio-respiratory illness, such as acute myocardial infarction, left heart decompensation or respiratory decompensation responsible for clinical presentation and symptoms and (iv) subsegmental PE as an incidental finding during diagnostic work-up for another suspected disease. All patients were followed for the in-hospital stay and one-year survival status was assessed by contacting the responsible registration offices.
      Diagnostic and therapeutic management was in accordance with the ESC 2008 (09/2008–08/2014) and 2014 (09/2014–03/2018) guidelines [
      • Torbicki A.
      • Perrier A.
      • Konstantinides S.
      • Agnelli G.
      • Galie N.
      • Pruszczyk P.
      • et al.
      Guidelines on the diagnosis and management of acute pulmonary embolism: the Task Force for the Diagnosis and Management of Acute Pulmonary Embolism of the European Society of Cardiology (ESC).
      ,
      • Konstantinides S.V.
      • Torbicki A.
      • Agnelli G.
      • Danchin N.
      • Fitzmaurice D.
      • Galie N.
      • et al.
      ESC guidelines on the diagnosis and management of acute pulmonary embolism.
      ] and local standard operating procedures. All related decisions were left to the discretion of the treating physicians and not influenced by the study protocol. Treating physicians were not informed about study results, thus any influence of the study on patient management or monitoring of treatment effects can be excluded. The study was conducted in accordance with the amended Declaration of Helsinki and was approved by the local independent Ethic Committee of the Medical University Göttingen, Germany (application number: 14/6/10); all patients gave informed written consent for participation in the study.
      Patients were stratified to risk classes according to the simplified Pulmonary Embolism Severity Index (sPESI) and the algorithm proposed by the 2019 ESC guidelines. [
      • Konstantinides S.V.
      • Meyer G.
      • Becattini C.
      • Bueno H.
      • Geersing G.J.
      • Harjola V.P.
      • et al.
      ESC Guidelines for the diagnosis and management of acute pulmonary embolism developed in collaboration with the European Respiratory Society (ERS).
      ] For calculation of all algorithms and scores, missing values were considered to be normal. [
      • Lankeit M.
      • Jimenez D.
      • Kostrubiec M.
      • Dellas C.
      • Hasenfuss G.
      • Pruszczyk P.
      • et al.
      Predictive value of the high-sensitivity troponin T assay and the simplified Pulmonary Embolism Severity Index in hemodynamically stable patients with acute pulmonary embolism: a prospective validation study.
      ] Tachycardia was defined as heart rate ≥100 beats per minute (bpm), hypotension as systolic blood pressure <90 mmHg and hypoxaemia as peripheral oxygen saturation <90%. Renal insufficiency was defined as a glomerular filtration rate (GFR) <60 ml/min/1.73 m2 body surface area. Active cancer was defined as known malignancy, treatment with antitumour therapy within the last 6 months, metastatic state or hematologic cancer that was not in complete remission. [
      • Raskob G.E.
      • van Es N.
      • Segers A.
      • Angchaisuksiri P.
      • Oh D.
      • Boda Z.
      • et al.
      Edoxaban for venous thromboembolism in patients with cancer: results from a non-inferiority subgroup analysis of the Hokusai-VTE randomised, double-blind, double-dummy trial.
      ] RV dysfunction on computed tomography pulmonary angiography (CTPA) was defined as right-to-left ventricular (RV/LV) diameter ratio ≥1.0. [
      • Konstantinides S.V.
      • Meyer G.
      • Becattini C.
      • Bueno H.
      • Geersing G.J.
      • Harjola V.P.
      • et al.
      ESC Guidelines for the diagnosis and management of acute pulmonary embolism developed in collaboration with the European Respiratory Society (ERS).
      ]
      The primary study outcome was an in-hospital adverse outcome, defined as PE-related death, cardiopulmonary resuscitation or vasopressor treatment. Further study outcomes included in-hospital and one-year all-cause mortality. Death was determined to be PE-related if either confirmed by autopsy or following a clinically severe episode of acute PE in absence of an alternative diagnosis. All events and causes of death were independently adjudicated by two of the authors (M.E. and C.S.) and disagreement was resolved by a third author (M.L.).

      2.2 Biomarker measurements

      Venous blood sampling was performed via peripheral venepuncture on admission or at the time of PE diagnosis as a part of routine clinical management. Routine venous blood gas analyses were performed using a standard point-of-care full blood gas analyses assay (GEM Premier 4000 analyser; Instrumentation Laboratory, Kirchheim, Germany). Plasma concentrations of high-sensitivity troponin T (hsTnT; Roche Diagnostics, Mannheim, Germany) were measured by the amedes MVZ wagnerstibbe laboratory in Göttingen, Germany and elevated concentrations were prospectively defined as ≥14 pg/ml. [
      • Lankeit M.
      • Jimenez D.
      • Kostrubiec M.
      • Dellas C.
      • Hasenfuss G.
      • Pruszczyk P.
      • et al.
      Predictive value of the high-sensitivity troponin T assay and the simplified Pulmonary Embolism Severity Index in hemodynamically stable patients with acute pulmonary embolism: a prospective validation study.
      ]

      2.3 Statistical analysis

      Categorical variables are presented as total numbers and percentages; continuous variables are presented as medians with interquartile ranges (IQR). Associations between binary and categorical variables were analysed using Fisher's exact test, Chi-square test or the Mantel-Haenszel test of trend, as appropriate. Spearman's rank correlation coefficient was used to test for statistical dependence of venous lactate from continuous variables. For comparison of continuous variables, the Mann-Whitney U test was employed.
      To investigate the prognostic performance of venous lactate levels with regard to study outcomes, receiver operating characteristic (ROC) curve analyses were performed to determine the area under the curve (AUC) with corresponding 95% confidence intervals (CIs). Youden index quantification was used to identify the optimal cut-off value for prediction of the primary study outcome.
      The prognostic performance of the established cut-off value for arterial lactate (2.0 mmol/l), the upper limit of normal for venous lactate (2.3 mmol/l) and the newly identified optimised cut-off value was expressed in terms of sensitivity, specificity, positive and negative predictive value and positive likelihood ratios. The prognostic value of the three lactate cut-off values and further parameters with regard to study outcomes was tested using univariable logistic regression analyses and results are presented as odds ratios (OR) with the corresponding 95% CI. To confirm the independent prognostic value of venous lactate, all predictors of study outcomes identified in univariate logistic regression analyses were entered in a multivariable logistic regression model with forward stepwise selection (inclusion criterion: p-value of the score test ≤5%; exclusion criterion: p-value of the likelihood-ratio test ≥10%).
      To investigate the incremental value of venous lactate in addition to the 2019 ESC risk assessment algorithm for prediction of adverse in-hospital outcomes, we performed a hierarchical binary logistic regression analysis. In step 1 the model included 2019 ESC risk classes only and in step 2 categorial information on venous lactate levels (group 1 = <2.3 mmol/l; group 2 = 2.3–3.2 mmol/l; group 3 = ≥3.3 mmol/l) were added. Likelihood-ratio chi², Nagelkerke pseudo-R², Hosmer-Lemeshow test and Harrell's c-statistic were calculated. To evaluate the potential benefit of a modified version of the 2019 ESC risk assessment algorithm that further stratified intermediate-risk patients based on venous lactate, we calculated user category net reclassification improvement (NRI) with the corresponding standard error. [
      • Pencina M.J.
      • D’Agostino Sr., RB
      • D’Agostino Jr., RB
      • Vasan R.S
      Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond.
      ] Kaplan-Meier analysis was used to compare the probability of one-year survival in subgroups stratified according to lactate levels at presentation; the log-rank test was used for comparison between groups.
      A two-sided significance level of α<0.05 was defined appropriate to indicate statistical significance. As this was an explorative testing, no adjustments for multiple testing were carried out. P-values were provided for descriptive reasons only and should be interpreted with caution and in connection with effect estimates. Statistical analysis was performed through Statistics Package for Social Sciences (IBM SPSS Statistics, Version 26, IBM Corp. Armonk, NY).

      3. Results

      Between September 2008 and March 2018, 851 patients were enrolled in PERGO. Exclusion criteria applied to (i) 279 (32.8%) patients with missing venous lactate measurements at presentation, (ii) 86 (10.1%) patients with high-risk PE, (iii) 25 (2.9%) patients with significant concomitant acute cardio-respiratory illness and (iv) 42 (4.9%) patients with subsegmental PE as an incidental finding. Hence, 419 (49.2%) patients were included in the present analysis.
      At presentation, 45 (10.7%) patients were classified as low-risk, 211 (50.4%) as intermediate-low-risk and 161 (38.4%) as intermediate-high-risk according to the 2019 ESC risk stratification algorithm. An in-hospital adverse outcome occurred in 24 (5.7%) patients. Overall, 17 (4.1%) patients died during the in-hospital stay; of those, 10 (58.8%) due to PE. Further information on comorbidities, initial presentation and outcomes is shown in Table 1, left column. A comparison of study patients and patients excluded due to missing lactate measurements is provided in Table s1 of the Online Supplement.
      Table 1Comorbidities, results from risk stratification and outcomes of study patients stratified to venous lactate levels.
      All patients (n = 419)Venous lactate ≥2.3 mmol/l (n = 119)p-value
      compared to patients with venous lactate <2.3 mmol/l.
      Venous lactate ≥3.3 mmol/l (n = 45)p-value
      compared to patients with venous lactate <3.3 mmol/l.
      Age ≥75 years160 (38.2%)53 (44.5%)0.0920 (44.4%)0.36
      Sex (female)220 (52.5%)64 (53.8%)0.7421 (46.7%)0.41
      Comorbidities
      Chronic heart failure60 (14.3%)20 (16.8%)0.369 (20%)0.25
      Coronary artery disease64 (15.3%)14 (11.8%)0.217 (15.6%)0.96
      Chronic pulmonary disease57 (13.6%)15 (12.6%)0.716 (13.3%)0.96
      Diabetes mellitus68 (16.2%)30 (25.2%)0.00215 (33.3%)<0.001
      Renal insufficiency142 (34.1%), n = 41750 (42%)0.0325 (55.6%)0.001
      Active cancer64 (15.3%), n = 41824 (20.3%), n = 1180.078 (17.8%)0.63
      Symptoms at presentation
      Dyspnea346 (83.2%), n = 41693 (78.2%)0.0832 (71.1%)0.022
      Syncope60 (14.4%), n = 41730 (25.2%)<0.00116 (35.6%)<0.001
      Clinical findings at presentation
      Tachycardia167 (40.1%), n = 41670 (58.8%)<0.00131 (68.9%)<0.001
      Hypoxaemia92 (23.2%), n = 39643 (37.1%), n = 116<0.00118 (41.9%), n = 430.002
      Laboratory and imaging markers
      hsTnT ≥14 pg/ml266 (66.7%), n = 39993 (83.8%), n = 111<0.00137 (86.0%), n = 430.004
      RV/LV diameter ratio ≥1.0 on CTPA245 (76.6%), n = 32080 (86.0%), n = 930.01130 (81.1%), n = 370.49
      2019 ESC risk stratification algorithm
      Low risk

      Intermediate-low risk

      Intermediate-high risk
      47 (11.2%)

      211 (50.4%)

      161 (38.4%)
      2 (1.7%)

      49 (41.2%)

      68 (57.1%)
      <0.0010 (0.0%)

      16 (35.6%)

      29 (64.4%)
      <0.001
      Outcome
      In-hospital adverse outcome

      Vasopressor treatment

      Cardiopulmonary resuscitation

      PE-related death
      24 (5.7%)

      18 (4.3%)

      7 (1.7%)

      10 (2.4%)
      16 (13.4%)

      13 (10.9%)

      5 (4.2%)

      5 (4.2%)
      <0.001

      <0.001

      0.011

      0.23
      12 (26.7%)

      10 (22.2%)

      4 (8.9%)

      4 (8.9%)
      <0.001

      <0.001

      <0.001

      0.002
      In-hospital all-cause mortality17 (4.1%)8 (5.6%)0.235 (11.1%)0.011
      Reperfusion treatment19 (4.5%)24 (16.9%)<0.0018 (17.8%)<0.001
      Statistically significant results are marked in bold letters.
      Abbreviations: HsTnT denotes high sensitivity troponin T; RV/LV, right/left ventricle; CTPA, computed tomography pulmonary angiography; ESC, European Society of Cardiology; PE, pulmonary embolism.Prognostic value of venous lactate.
      a compared to patients with venous lactate <2.3 mmol/l.
      b compared to patients with venous lactate <3.3 mmol/l.
      The median venous lactate concentration on admission was 1.6 (IQR 1.2–2.4) mmol/l. Venous lactate concentrations showed a positive correlation with heart rate (r = 0.32, p<0.001) and an inverse correlation with systolic blood pressure (r=−0.13, p = 0.008).
      Patients who suffered an in-hospital adverse outcome had higher venous lactate concentrations than patients with a favourable clinical course (3.1 [IQR 1.3–4.9] mmol/l vs. 1.6 [IQR 1.2–2.3] mmol/l, p = 0.001). The rate of in-hospital adverse outcomes was associated with increasing lactate concentrations at presentation (p <0.001 for trend; Fig. 1).
      Fig. 1
      Fig. 1Rate of an in-hospital adverse outcome stratified to venous lactate level at presentation.
      Using ROC analysis, we calculated an AUC of 0.70 (95% CI 0.57–0.84) for the prediction of an in-hospital adverse outcome and identified a venous lactate concentration of 3.3 mmol/l as the optimal cut-off value. Of note, Youden's indices for the cut-off values 2.3 mmol/l and 3.3 mmol/l were almost identical (0.41 vs. 0.42). A comparison of test characteristics for the optimal cut-off value and prespecified cut-off values is presented in Table 2. Due to the superior specificity provided by the 2.3 mmol/l and 3.3 mmol/l cut-off values compared to 2.0 mmol/l, all further analyses in the main manuscript are based on these two values. Moreover, results of risk stratification based on the 2.0 mmol/l cut-off value are provided in the Online Supplement.
      Table 2Prognostic performance of different venous lactate cut-off values with regard to (A) in-hospital adverse outcome and (B) all-cause mortality.
      A: In-hospital adverse outcome
      PrevalenceIn-hospital adverse outcome rateSensitivity  (95% CI)Specificity (95% CI)PPV (95% CI)NPV (95% CI)LR+OR  (95% CI)
      Venous lactate  ≥2.0 mmol/l37.2%10.3%66.7% (44.7–83.6)64.6% (59.6–69.2)0. 10  (0.06–0.16)0.97  (0.94–0.99)1.93.64 (1.52–8.73)
      Venous lactate  ≥2.3 mmol/l28.4%13.6%66.7% (44.7–83.6)73.9%  (69.2–78.1)0.13  (0.08–021)0.97  (0.95–0.99)2.65.67  (2.36–13.64)
      Venous lactate  ≥3.3 mmol/l10.7%26.7%50.0%  (29.6–70.4)91.6%  (88.4–94.1)0.27  (0.16–0.41)0.97  (0.94–0.98)6.010.97  (4.57–26.34)
      B: In-hospital all-cause mortality
      PrevalenceIn-hospital mortality rateSensitivity  (95% CI)Specificity (95% CI)PPV (95% CI)NPV (95% CI)LR+OR  (95% CI)
      Venous lactate  ≥2.0 mmol/l37.2%4.5%41.2%  (19.4–66.5)62.9%  (58.0–67.6)0.04  (0.02–0.09)0.96  (0.93–0.98)1.11.19  (0.44–3.19)
      Venous lactate  ≥2.3 mmol/l28.4%5.9%41.2% (19.4–66.5)72.1% (67.4–76.4)0.06  (0.03–0.12)0.97  (0.94–0.98)1.51.81  (0.67–4.48)
      Venous lactate  ≥3.3 mmol/l10.7%11.1%29.4%  (11.4–56.0)90.0%  (86.6–92.7)0.11  (0.05–0.23)0.97  (0.94–0.98)3.03.77 (1.26–11.25)
      Statistically significant results are marked in bold letters.
      Abbreviations: CI denotes confidence interval; PPV, positive predictive value; NPV, negative predictive value; LR+, positive likelihood ratio; OR, odds ratio.
      Baseline characteristics and results of risk stratification in patients stratified to venous lactate cut-off values of 2.3 mmol/l and 3.3 mmol/l are provided in Table 1, middle and right columns. Patients with elevated venous lactate more frequently presented with syncope and had higher rates of tachycardia, hypoxaemia, elevated troponin levels and signs of RV dysfunction on CTPA.
      Venous lactate ≥2.3 mmol/l was observed in 119 (28.4%) patients and predicted an in-hospital adverse outcome (OR 5.7 [95% CI 2.4–13.6]; Table 2) but not all-cause mortality (OR 1.8 [95% CI 0.7–4.9]). Venous lactate ≥3.3 mmol/l was observed in 45 (10.7%) patients and predicted both, an in-hospital adverse outcome (OR 11.0 [95% CI 4.6–26.3]) and all-cause mortality (OR 3.8 [95% CI 1.3–11.3]; Table 2). When entered in multivariable models that included all outcome predictors identified based on the results of univariate logistic regression analyses, the independent prognostic value of the two investigated cut-off values with regard to the study outcomes was confirmed (Table 3).
      Table 3Predictors of (A) in-hospital adverse outcome and (B) all-cause mortality identified using univariable and multivariable logistic regression models.
      A: In-hospital adverse outcome
      Univariable modelMultivariable model 1Multivariable model 2
      OR (95% CI)p-valueOR (95% CI)p-valueOR (95% CI)p-value
      Renal insufficiency2.90 (1.25–6.7)0.010.130.36
      Syncope3.51 (1.42–8.68)0.0040.150.30
      Venous lactate ≥2.3 mmol/l5.67 (2.36–13.64)<0.0015.10 (2.01–12.98)0.001Not included
      Venous lactate ≥3.3 mmol/l10.97 (4.57–26.34)<0.001Not included9.68 (3.89–24.10)<0.001
      hsTnT ≥14 pg/ml12.49 (1.67–93.55)0.0028.03 (1.05–61.54)0.0458.67 (1.13–66.49)0.038
      B: In-hospital all-cause mortality
      Univariable modelMultivariable model 3
      OR (95% CI)p-valueOR (95% CI)p-value
      Active cancer3.23 (1.15–9.06)0.0193.02 (1.05–9.85)0.041
      Venous lactate ≥3.3 mmol/l3.77 (1.26–11.25)0.0113.20 (1.04–9.85)0.042
      hsTnT ≥14 pg/ml8.45 (1.11–64.4)0.0146.98 (0.90–53.83)0.06
      Statistically significant results are marked in bold letters.
      Abbreviations: OR denotes odds ratio; CI, confidence interval; hsTnT, high sensitivity troponin T.
      Data on one-year mortality of patients stratified according to venous lactate levels at presentation are provided in the Online Supplement.

      3.1 Venous lactate for risk stratification

      If venous lactate was added to the 2019 ESC algorithm, the fit of a hierarchical binary logistic regression model (including 2019 ESC risk classes [step 1] and categories of venous lactate elevation [step 2]) improved (chi2 = 18.17; p <0.001), and results of the Hosmer-Lemeshow test remained non-significant for both models (p = 0.498 and p = 0.322, respectively). The overall fit of the model assessed using Nagelkerke pseudo-R2 increased from 0.086 to 0.202 and the c-index improved from 0.69 (95% CI 0.59–0.79) to 0.82 (95% CI 0.74–0.89). Further, as shown in Fig. 2 and Figure s3 of the Online Supplement, venous lactate concentrations ≥2.3 mmol/l identified patients with higher rates of an adverse outcome in intermediate-low risk patients (p <0.001) and provided an OR for adverse outcome prediction of 22.5 (95% CI 2.6–191.6). Conversely, venous lactate <2.3 mmol/l had a negative predictive value for the occurrence of an adverse outcome of 0.99 (95% CI 0.97–1.00), resulting in a net reclassification improvement (NRI; 0.65±0.16, p <0.001). In the intermediate-high risk group, venous lactate ≥3.3 mmol predicted adverse outcomes with an OR of 5.2 (95% CI 1.8–15.0), but NRI slightly failed to reach statistical significance (0.32±0.17, p = 0.06).
      Fig. 2
      Fig. 2Venous lactate for risk stratification in addition to the 2019 ESC algorithm Abbreviations: PE denotes pulmonary embolism; ESC, European Society of Cardiology.

      4. Discussion

      In the present study, we evaluated the prognostic value of venous lactate in normotensive PE patients. Our findings obtained in 419 patients can be summarised as follows: (i) venous lactate above the upper limit of normal (2.3 mmol/l) was associated with an increased risk for an in-hospital adverse outcome, (ii) lactate concentrations exceeding an optimised cut-off value of 3.3 mmol/l were predictive of both an in-hospital adverse outcome and all-cause mortality, (iii) the previously proposed cut-off value for arterial lactate of 2.0 mmol/l has low prognostic value in venous samples and should not be used and (iv) information on venous lactate added to the 2019 ESC algorithm may further improve risk stratification of normotensive PE patients.
      Haemodynamic instability due to RV failure is the main reason for adverse outcomes in PE patients. Accordingly, risk stratification of acute PE in based on signs indicating myocardial dysfunction or damage, e.g. RV dilation on diagnostic imaging and elevated troponin levels. [
      • Konstantinides S.V.
      • Meyer G.
      • Becattini C.
      • Bueno H.
      • Geersing G.J.
      • Harjola V.P.
      • et al.
      ESC Guidelines for the diagnosis and management of acute pulmonary embolism developed in collaboration with the European Respiratory Society (ERS).
      ] However, these markers provide only indirect information on the adequacy of cardiac output, the critical determinant of haemodynamic impairment. Direct assessment of peripheral (hypo)perfusion using plasma lactate measurements might therefore be a more specific indicator of threatening haemodynamic instability.
      This concept is supported by previous investigations that investigated the role of arterial lactate for outcome prediction in both, unselected and normotensive patients with PE. [
      • Vanni S.
      • Socci F.
      • Pepe G.
      • Nazerian P.
      • Viviani G.
      • Baioni M.
      • et al.
      High plasma lactate levels are associated with increased risk of in-hospital mortality in patients with pulmonary embolism.
      ,
      • Vanni S.
      • Viviani G.
      • Baioni M.
      • Pepe G.
      • Nazerian P.
      • Socci F.
      • et al.
      Prognostic value of plasma lactate levels among patients with acute pulmonary embolism: the thrombo-embolism lactate outcome study.
      ,
      • Vanni S.
      • Jimenez D.
      • Nazerian P.
      • Morello F.
      • Parisi M.
      • Daghini E.
      • et al.
      Short-term clinical outcome of normotensive patients with acute PE and high plasma lactate.
      ,
      • Vanni S.
      • Nazerian P.
      • Bova C.
      • Bondi E.
      • Morello F.
      • Pepe G.
      • et al.
      Comparison of clinical scores for identification of patients with pulmonary embolism at intermediate-high risk of adverse clinical outcome: the prognostic role of plasma lactate.
      ,
      • Freitas P.
      • Santos A.R.
      • Ferreira A.M.
      • Oliveira A.
      • Goncalves M.
      • Corte-Real A.
      • et al.
      Derivation and external validation of the SHIeLD score for predicting outcome in normotensive pulmonary embolism.
      ] In a cohort of 287 unselected PE patients, Vanni et al. identified an arterial lactate cut-off value of 2.0 mmol/l to predict in-hospital mortality with an OR of 4.6 (95% CI 1.4–17.6). [
      • Vanni S.
      • Socci F.
      • Pepe G.
      • Nazerian P.
      • Viviani G.
      • Baioni M.
      • et al.
      High plasma lactate levels are associated with increased risk of in-hospital mortality in patients with pulmonary embolism.
      ] This cut-off value was later validated in 270 unselected PE patients included the prospective Thrombo Embolism Lactate Outcome Study (TELOS). [
      • Vanni S.
      • Viviani G.
      • Baioni M.
      • Pepe G.
      • Nazerian P.
      • Socci F.
      • et al.
      Prognostic value of plasma lactate levels among patients with acute pulmonary embolism: the thrombo-embolism lactate outcome study.
      ] In a cohort of normotensive PE patients, arterial lactate ≥2.0 mmol/l increased the risk of developing PE-related complications (OR 6.9 [95% CI 2.6–18.2]) and 30-day all-cause mortality (OR 2.5; 95% CI 1.1–5.5). [
      • Vanni S.
      • Jimenez D.
      • Nazerian P.
      • Morello F.
      • Parisi M.
      • Daghini E.
      • et al.
      Short-term clinical outcome of normotensive patients with acute PE and high plasma lactate.
      ] Finally, information on arterial lactate has been integrated in different risk assessment strategies such as the expanded BOVA score and the SHIelD score for improved risk prediction in intermediate-risk PE patients. [
      • Vanni S.
      • Nazerian P.
      • Bova C.
      • Bondi E.
      • Morello F.
      • Pepe G.
      • et al.
      Comparison of clinical scores for identification of patients with pulmonary embolism at intermediate-high risk of adverse clinical outcome: the prognostic role of plasma lactate.
      ,
      • Freitas P.
      • Santos A.R.
      • Ferreira A.M.
      • Oliveira A.
      • Goncalves M.
      • Corte-Real A.
      • et al.
      Derivation and external validation of the SHIeLD score for predicting outcome in normotensive pulmonary embolism.
      ]
      Unfortunately, the clinical applicability of these uniformly positive results is limited by the requirement of obtaining an arterial blood sample. An arterial puncture demands additional time, requires special training of the personnel and subjects patients to pain and inconvenience. [
      • Kruse O.
      • Grunnet N.
      • Barfod C.
      Blood lactate as a predictor for in-hospital mortality in patients admitted acutely to hospital: a systematic review.
      ] Furthermore, arterial punctures present an additional bleeding risk in patients treated with thrombolytic therapy due to haemodynamic worsening.
      To overcome this important limitation, the present study investigated whether a similar prognostic value is provided by lactate measured from peripheral venous samples, that can be easily obtained during a routine (venous) blood draw. Our results confirm the study hypothesis. Even modest venous lactate elevations above the upper limit of normal (2.3 mmol/l) increased the odds of an in-hospital adverse outcome. Optimal prognostic performance was observed for a venous lactate cut-off value of 3.3 mmol/l, that predicted adverse outcomes as well as all-cause mortality.
      Even though peripheral venous and arterial lactate concentrations are highly correlated, venous levels are on average about 0.2 to 0.3 mmol/l higher compared to arterial concentrations. [
      • Kruse O.
      • Grunnet N.
      • Barfod C.
      Blood lactate as a predictor for in-hospital mortality in patients admitted acutely to hospital: a systematic review.
      ,
      • Mikami A.
      • Ohde S.
      • Deshpande G.A.
      • Mochizuki T.
      • Otani N.
      • Ishimatsu S.
      Can we predict arterial lactate from venous lactate in the ED?.
      ] This might explain why the previously identified arterial cut-off value (2.0 mmol/l) [
      • Vanni S.
      • Socci F.
      • Pepe G.
      • Nazerian P.
      • Viviani G.
      • Baioni M.
      • et al.
      High plasma lactate levels are associated with increased risk of in-hospital mortality in patients with pulmonary embolism.
      ], that lies within the normal range for venous samples, had lower specificity compared to the other investigated cut-off values (Table 2). Thus, it should not be used when interpreting venous lactate for risk stratification purposes.
      Venous lactate adds to the predictive value of the 2019 ESC risk assessment algorithm. In intermediate-low-risk patients, non-elevated venous lactate (<2.3 mmol/l) excluded the occurrence of an adverse outcome with a negative predictive value of more than 99%, while in intermediate-high-risk patients venous lactate ≥3.3 mmol/l identified a subgroup at increased risk for developing adverse outcomes (Fig. 2).

      4.1 Limitations

      Of note, a considerable number of patients were excluded from the study due to missing lactate measurements at presentation. However, the comparison of patients with and without lactate measurements provided in Table s1 did not reveal relevant differences between the two groups. The small number of patients with an adverse outcome (5.7%) and with a venous lactate concentration ≥3.3 mmol/l (10.7%) may have impaired the ability to detect statistically significant differences between subgroups. Furthermore, the single-center design limits the generalizability of our findings.

      4.2 Conclusion

      Our results confirm the prognostic value of peripheral venous lactate in normotensive patients with acute PE. An increased risk for an in-hospital adverse outcome was observed in all patients with venous lactate concentrations exceeding the upper limit of normal (≥2.3 mmol/l) and a cut-off value of 3.3 mmol/l provided optimal prognostic performance predicting both, an in-hospital adverse outcome and all-cause mortality. If added to the 2019 ESC algorithm, information on venous lactate may further improve risk stratification of intermediate-risk patients.

      5. Funding

      This study was supported by the German Federal Ministry of Education and Research (BMBF 01EO1503). The authors are responsible for the contents of this publication.
      CNIC is supported by the Spanish Ministry of Economy and Competitiveness (MINECO) and the Pro-CNIC Foundation, and is a Severo Ochoa Center of Excellence (MINECO award SEV-2015–0505).

      Authors' contributions

      Matthias Ebner
      • acquisition, analysis and interpretation of the data,
      • drafting of the manuscript and revising the manuscript critically for important intellectual content,
      • final approval of the manuscript to be published,
      • agreement to be accountable for all aspects of the work
      Charlotta F. Pagel
      • acquisition and interpretation of the data,
      • revising the manuscript critically for important intellectual content,
      • final approval of the manuscript to be published,
      • agreement to be accountable for all aspects of the work
      Carmen Sentler
      • acquisition and interpretation of the data
      • revising the manuscript critically for important intellectual content,
      • final approval of the manuscript to be published,
      • agreement to be accountable for all aspects of the work
      Veli-Pekka Harjola
      • analysis and interpretation of the data,
      • revising the manuscript critically for important intellectual content,
      • final approval of the manuscript to be published,
      • agreement to be accountable for all aspects of the work
      Héctor Bueno
      • analysis and interpretation of the data,
      • revising the manuscript critically for important intellectual content,
      • final approval of the manuscript to be published,
      • agreement to be accountable for all aspects of the work
      Markus Lerchbaumer
      • acquisition and interpretation of the data
      • revising the manuscript critically for important intellectual content,
      • final approval of the manuscript to be published,
      • agreement to be accountable for all aspects of the work
      Karl Stangl
      • interpretation of the data,
      • revising the manuscript critically for important intellectual content,
      • final approval of the manuscript to be published,
      • agreement to be accountable for all aspects of the work
      Burkert Pieske
      • interpretation of the data,
      • revising the manuscript critically for important intellectual content,
      • final approval of the manuscript to be published,
      • agreement to be accountable for all aspects of the work
      Gerd Hasenfuß
      • interpretation of the data
      • revising the manuscript critically for important intellectual content,
      • final approval of the manuscript to be published,
      • agreement to be accountable for all aspects of the work
      Stavros Konstantinides
      • interpretation of the data
      • revising the manuscript critically for important intellectual content,
      • final approval of the manuscript to be published,
      • agreement to be accountable for all aspects of the work
      Mareike Lankeit
      • conception and design of the study,
      • acquisition, analysis and interpretation of the data,
      • drafting of the manuscript and revising the manuscript critically for important intellectual content,
      • final approval of the manuscript to be published,
      • agreement to be accountable for all aspects of the work

      Declaration of competing interest

      None.

      Acknowledgements

      The authors thank Dr. Christian Thode and Daniela Brehm (amedes MVZ wagnerstibbe, Göttingen, Germany) for performing the biomarker measurements. This research was supported by the UMG Biobank as a core facility of the University Medical Center Göttingen, Germany by provision of quality assured biomaterial. This publication is part of the medical doctoral thesis of Charlotta Friederike Pagel.

      Appendix. Supplementary materials

      References

        • Raskob G.E.
        • Angchaisuksiri P.
        • Blanco A.N.
        • Buller H.
        • Gallus A.
        • Hunt B.J.
        • et al.
        Thrombosis: a major contributor to global disease burden.
        Arterioscler Thromb Vasc Biol. 2014; 34: 2363-2371
        • Konstantinides S.V.
        • Meyer G.
        • Becattini C.
        • Bueno H.
        • Geersing G.J.
        • Harjola V.P.
        • et al.
        ESC Guidelines for the diagnosis and management of acute pulmonary embolism developed in collaboration with the European Respiratory Society (ERS).
        Eur Heart J. 2020; 41: 543-603
        • Harjola V.P.
        • Mebazaa A.
        • Celutkiene J.
        • Bettex D.
        • Bueno H.
        • Chioncel O.
        • et al.
        Contemporary management of acute right ventricular failure: a statement from the Heart Failure Association and the Working Group on Pulmonary Circulation and Right Ventricular Function of the European Society of Cardiology.
        Eur J Heart Fail. 2016; 18: 226-241
        • Thiele H.
        • Ohman E.M.
        • Desch S.
        • Eitel I.
        • de Waha S.
        Management of cardiogenic shock.
        Eur Heart J. 2015; 36: 1223-1230
        • Regnier M.A.
        • Raux M.
        • Le Manach Y.
        • Asencio Y.
        • Gaillard J.
        • Devilliers C.
        • et al.
        Prognostic significance of blood lactate and lactate clearance in trauma patients.
        Anesthesiology. 2012; 117: 1276-1288
        • Casserly B.
        • Phillips G.S.
        • Schorr C.
        • Dellinger R.P.
        • Townsend S.R.
        • Osborn T.M.
        • et al.
        Lactate measurements in sepsis-induced tissue hypoperfusion: results from the Surviving Sepsis Campaign database.
        Crit Care Med. 2015; 43: 567-573
        • Vanni S.
        • Socci F.
        • Pepe G.
        • Nazerian P.
        • Viviani G.
        • Baioni M.
        • et al.
        High plasma lactate levels are associated with increased risk of in-hospital mortality in patients with pulmonary embolism.
        Acad Emerg Med. 2011; 18: 830-835
        • Vanni S.
        • Viviani G.
        • Baioni M.
        • Pepe G.
        • Nazerian P.
        • Socci F.
        • et al.
        Prognostic value of plasma lactate levels among patients with acute pulmonary embolism: the thrombo-embolism lactate outcome study.
        Ann Emerg Med. 2013; 61: 330-338
        • Vanni S.
        • Jimenez D.
        • Nazerian P.
        • Morello F.
        • Parisi M.
        • Daghini E.
        • et al.
        Short-term clinical outcome of normotensive patients with acute PE and high plasma lactate.
        Thorax. 2015; 70: 333-338
        • Vanni S.
        • Nazerian P.
        • Bova C.
        • Bondi E.
        • Morello F.
        • Pepe G.
        • et al.
        Comparison of clinical scores for identification of patients with pulmonary embolism at intermediate-high risk of adverse clinical outcome: the prognostic role of plasma lactate.
        Intern Emerg Med. 2017; 12: 657-665
        • Kruse O.
        • Grunnet N.
        • Barfod C.
        Blood lactate as a predictor for in-hospital mortality in patients admitted acutely to hospital: a systematic review.
        Scand J Trauma Resusc Emerg Med. 2011; 19: 74
        • Younger J.G.
        • Falk J.L.
        • Rothrock S.G.
        Relationship between arterial and peripheral venous lactate levels.
        Acad Emerg Med. 1996; 3: 730-734
        • Mikami A.
        • Ohde S.
        • Deshpande G.A.
        • Mochizuki T.
        • Otani N.
        • Ishimatsu S.
        Can we predict arterial lactate from venous lactate in the ED?.
        Am J Emerg Med. 2013; 31: 1118-1120
        • Hellenkamp K.
        • Pruszczyk P.
        • Jimenez D.
        • Wyzgal A.
        • Barrios D.
        • Ciurzynski M.
        • et al.
        Prognostic impact of copeptin in pulmonary embolism: a multicentre validation study.
        Eur Respir J. 2018; 51: 1702037
        • Ebner M.
        • Kresoja K.P.
        • Keller K.
        • Hobohm L.
        • Rogge N.I.J.
        • Hasenfuss G.
        • et al.
        Temporal trends in management and outcome of pulmonary embolism: a single-centre experience.
        Clin Res Cardiol. 2020; 109: 67-77
        • Torbicki A.
        • Perrier A.
        • Konstantinides S.
        • Agnelli G.
        • Galie N.
        • Pruszczyk P.
        • et al.
        Guidelines on the diagnosis and management of acute pulmonary embolism: the Task Force for the Diagnosis and Management of Acute Pulmonary Embolism of the European Society of Cardiology (ESC).
        Eur Heart J. 2008; 29: 2276-2315
        • Konstantinides S.V.
        • Torbicki A.
        • Agnelli G.
        • Danchin N.
        • Fitzmaurice D.
        • Galie N.
        • et al.
        ESC guidelines on the diagnosis and management of acute pulmonary embolism.
        Eur Heart J. 2014; 35 (201469a-69k): 3033-3069
        • Lankeit M.
        • Jimenez D.
        • Kostrubiec M.
        • Dellas C.
        • Hasenfuss G.
        • Pruszczyk P.
        • et al.
        Predictive value of the high-sensitivity troponin T assay and the simplified Pulmonary Embolism Severity Index in hemodynamically stable patients with acute pulmonary embolism: a prospective validation study.
        Circulation. 2011; 124: 2716-2724
        • Raskob G.E.
        • van Es N.
        • Segers A.
        • Angchaisuksiri P.
        • Oh D.
        • Boda Z.
        • et al.
        Edoxaban for venous thromboembolism in patients with cancer: results from a non-inferiority subgroup analysis of the Hokusai-VTE randomised, double-blind, double-dummy trial.
        Lancet Haematol. 2016; 3: e379-e387
        • Pencina M.J.
        • D’Agostino Sr., RB
        • D’Agostino Jr., RB
        • Vasan R.S
        Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond.
        Stat Med. 2008; 27: 157-172
        • Freitas P.
        • Santos A.R.
        • Ferreira A.M.
        • Oliveira A.
        • Goncalves M.
        • Corte-Real A.
        • et al.
        Derivation and external validation of the SHIeLD score for predicting outcome in normotensive pulmonary embolism.
        Int J Cardiol. 2019; 281: 119-124