Advertisement

Clinical practice guidelines adaptation for internists - An EFIM methodology

      Highlights

      • We aim to adapt guidelines to every day Internal Medicine practice
      • Good guidelines published during last 5 years will be preferably selected
      • Some summaries could only remark lack of evidence for aged patients

      Abstract

      The rising number of clinical guidelines poses a new challenge to the internists. The main problems are: 1) available documents suffer from heterogeneous methodological quality, and 2) most of clinical guidelines target an 'ideal' patient affected by a single condition, while in real practice internists must face with comorbid patients typically undergoing a polypharmacy.
      To help address this challenge, EFIM Clinical Practice Working Group started a project aimed to answer a series of relevant clinical questions, by selecting the best available guidance containing recommendations applicable to complex patients under polypharmacy. The project started with the creation of a research protocol containing details about all the steps needed to write the Clinical Practice Guideline Summary. In particular, this methodological document specifies the rules: 1) to select topics and clinical questions; 2) to build up a panel of experts, carefully managing eventual conflict of interests; 3) to critically appraise clinical guidelines (using a validated tool as AGREE II), selecting the most valid and applicable to the common clinical practice (using ADAPTE; 4) to address and solve potential disagreements among the selected documents.

      Keywords

      Introduction

      With the easy access to internet, the increasing number of clinical studies and the better quality of these studies, the main challenge for the clinicians nowadays is to quickly find a relevant evidence and efficiently assess its quality. That is why the clinical practice guidelines (CPG) which summarize evidence and give a strict advice what to do in particular clinical circumstances continuously gain reputation and popularity. As a consequence the number of CPG produced every year is increasing. Although the quality of guidelines is much better now [
      • Seto K
      • Matsumoto K
      • Fujita S
      • Kitazawa T
      • Amin R
      • Hatakeyama Y
      • et al.
      Quality assessment of clinical practice guidelines using the AGREE instrument in Japan: a time trend analysis.
      ,
      • Zhou H
      • Zhang S
      • Sun X
      • Yang D
      • Zhuang X
      • Guo Y
      • et al.
      Lipid management for coronary heart disease patients: an appraisal of updated international guidelines applying appraisal of guidelines for research and evaluation II-clinical practice guideline appraisal for lipid management in coronary heart disease.
      ] than in previous decades thanks to the developed evidence based medicine (EBM) methodologies like e.g. GRADE [
      The GRADE Working Group
      Grading quality of evidence and strength of recommendations.
      ,
      • Guyatt G
      • Oxman AD
      • Akl EA
      • Kunz R
      • Vist G
      • Brozek J
      • et al.
      GRADE guidelines: 1. Introduction: GRADE evidence profiles and summary of findings tables.
      ], there are still many documents which are of poor quality and biased [
      • Alexander PE
      • Brito JP
      • Neumann I
      • Gionfriddo MR
      • Bero L
      • Djulbegovic B
      • et al.
      World Health Organization strong recommendations based on low-quality evidence (study quality) are frequent and often inconsistent with GRADE guidance.
      ,
      • Li Q
      • Li X
      • Wang J
      • Liu H
      • Kwong JS
      • Chen H
      • et al.
      Diagnosis and treatment for hyperuricemia and gout: a systematic review of clinical practice guidelines and consensus statements.
      ]. Another problem pointed out by European Federation of Internal Medicine (EFIM) is the lack of CPG specifically targeted at internal medicine doctors. Most of existing guidelines are created by specialists in partial fields of internal medicine (e.g. cardiologists, pneumonologists, rheumathologists, nephrologists etc.). Therefore most of guidelines do not address the problems which are very common in internal medicine practice e.g. multimorbidity and polypharmacy [
      • Jansen J
      • McKinn S
      • Bonner C
      • Les Irwig
      • Doust J
      • Glasziou P
      • et al.
      Systematic review of clinical practice guidelines recommendations about primary cardiovascular disease prevention for older adults.
      ]. These issues are of particular importance in the population of elderly patients. About half of people over 65 years of age have at least three coexisting chronic conditions and one in five has five or more [
      • Marengoni A
      • Winblad B
      • Karp A
      • Fratiglioni L
      Prevalence of chronic diseases and multimorbidity among the elderly population in Sweden.
      ,
      • Barnett K
      • Mercer SW
      • Norbury M
      • Watt G
      • Wyke S
      • Guthrie B
      Epidemiology of multimorbidity and implications for health care, research, and medical education: a crosssectional study.
      ]. The usual single disease approach and adherence to all CPG related to each illness make polypharmacy and inappropriate medication use in such patients inevitable [
      • Mangin D
      • Bahat G
      • Golomb BA
      • Mallery LH
      • Moorhouse P
      • Onder G
      • et al.
      International group for reducing inappropriate medication use & polypharmacy (IGRIMUP): position statement and 10 recommendations for action.
      ,
      • O'Mahony D
      • Curtin D.
      Deprescribing in older people: why it matters in routine clinical practice.
      ]. Whereas in some situations older patients do not receive effective treatment [
      • Van de Werf F.
      Reperfusion treatment in acute myocardial infarction in elderly patients.
      ]. According to International Group for Reducing Inappropriate Medication Use & Polypharmacy (IGRIMUP) Position Statement: one physician should co-ordinate decisions within a shared framework (preferably in a generalist setting suited to consideration of multimorbidity and polypharmacy ideally with a pharmacist partner) [
      • Mangin D
      • Bahat G
      • Golomb BA
      • Mallery LH
      • Moorhouse P
      • Onder G
      • et al.
      International group for reducing inappropriate medication use & polypharmacy (IGRIMUP): position statement and 10 recommendations for action.
      ]. An internist seems as a good candidate to take this role.
      Additionally, even if recommendations related to internal medicine practice are present in the documents created by specialists they are sometimes not widely disseminated e.g. presented in supplementary materials.
      Therefore it is needed to develop a network that will provide the CPG documents for internists taking the above mentioned problems into account. As developing unbiased CPG with the use of GRADE methodology (currently considered as the most valid methodology to create CPG) needs a lot of resources and is time consuming it was chosen by EFIM to adapt the best available evidence based guidelines instead of creating the new ones from scratch. Therefore the EFIM CPG Working Group (CPG-WG) was constituted. Each EFIM member (i.e. national society of internal medicine) was asked to delegate a representative who is experienced in CPG methodology and/or development. The group is composed of experienced internal medicine clinicians/researchers, some members work in international CPG working groups (e.g. European Society of Cardiology) and some EBM/CPG methodologists. EFIM CPG-WG aims to develop the methodology to quickly adapt the existing valid CPG and then to guide and coordinate the process of producing at least 2 documents per year. This group will also be responsible to prioritize the CPG topics with the help of external experts. In this paper the methodology of the adaptation is presented. The main aim of the process is to get a reliable product dedicated to internal medicine doctors which will be easy to implement in everyday clinical practice and will fit the best the needs of their patients. It is believed that the CPG developed and disseminated by EFIM will be broadly used by European internists and this will close the existing gap between the guidelines and the management of the patients [
      • Ena J
      • Gómez-Huelgas R
      • Romero-Sánchez M
      • Gaviria AZ
      • Calzada-Valle A
      • Varela-Aguilar JM
      • et al.
      Diabetes and obesity working group of the spanish society of internal medicine. Hyperglycemia management in patients admitted to internal medicine in Spain: a point-prevalence survey examining adequacy of glycemic control and guideline adherence.
      ].

      2. Methodology

      The EFIM CPG-WG during the first meeting chose 3 members (IM, WL, LM) who would be responsible to elaborate the methodology which would be used to prepare EFIM guidelines summary for internists. It was emphasized that the adaptation process needs to allow the quick development of 2 documents per year. Therefore the methods group decided to use the existing tools (AGREE [
      Appraisal of Guidelines, Research, and Evaluation in Europe (AGREE) Collaborative Group
      For the AGREE collaborative group. Guideline development in Europe: an international comparison.
      ,] and ADAPTE []), simplifying and adapting them to the needs of the WG. The methods group in iterative rounds of consultations refined the steps and items proposed by both existing tools with the purpose to define a feasible and reproducible adaptation procedure. The suggested methodology was accepted by all WG members. The adaptation process of CPG for European internists is described below (see also Fig. 1).

      2.1 The documents to be delivered

      Each CPG document is expected to be short with a few recommendations related to internal medicine clinical practice, gathered from reliable and updated CPG. It will be called Clinical Practice Guidelines Summary for Internal Medicine.

      2.2 Selection of the topics

      The EFIM CPG-WG will choose the topics based on brainstorming and then internal selection among CPG-WG during their meetings.
      Additionally in future to make the process more systematic, the national societies will be asked to list their priority topics (this does not apply to first 2 documents which are already under preparation). They will be encouraged to propose a set of clinical problems where e.g. the coincidence of comorbidity could result in misleading or controversial recommendations.

      2.3 Selection of the elaborating team for the adaptation task

      CPG-WG members will appoint a task force for each Guidelines Adaptation task according to the following structure: the chairperson (CPG-WG member), 2-3 CPG-WG members, 2-3 external experts in the specific field/topic. Task force members belonging to CPG-WG will propose the external experts and select them by voting.
      It is recommended to have an information specialist in the task force who will be responsible for guideline searches in various sources (NICE; SIGN; International Medical Societies, MEDLINE). It is suggested that the task force members have some knowledge about CPG (developing process, grading recommendations, etc).

      2.4 Selection of the clinical questions

      The clinical questions (PICOs) should be formulated by the national representatives in the CPG-WG. Each CPG-WG member can suggest a list of up to 5 PICOs based on their clinical relevance to internal medicine practice. An appointed person from the task force combines these lists together. The PICOs which were proposed most frequently should be chosen (with no more than 10 in total). The list of PICOs should be agreed by all task force members. In case of disagreement, the chairperson has the final vote.

      2.5 Selection of CPG used for adaptation?

      The relevant CPG related to the topic proposed by the panel will be searched, preferably by an information specialist (a task force member) in available sources. No limits will be applied in the first search. If guidelines published in the last 5 years are identified they will be preferably chosen. The older documents will be referred to if the new ones are of poor quality and/or do not include the recommendations (PICOs) of interest. The quality of existing CPG that recommendations are taken from will be assessed using AGREE-II Instrument []. The guidelines will be appraised by 3-4 task force members using the AGREE score for Domain 1 (Scope and Purpose: Items 1-3) and Domain 3 (Rigour of Development Items 7-14). The mean of their scores will be calculated for each item and each domain. A CPG will be selected if:
      • a)
        the mean score is at least 3 for each item (i.e. at least 9 in Domain 1 AND at least 24 in Domain 3)
      • b)
        at least 50% threshold for each of these domains [calculated as follow: (Obtained score – Minimum possible score)/(Maximum possible score – Minimum possible score) x 100% is reached.
      It has been emphasized that caution is needed if the identified original CPG is older than 5 years. In this case, the recommendations can be included only if there are no other more recent CPG relevant to the PICO.

      2.6 Selection of recommendations from the original existing CPG

      The answer for each PICO will be sought in the selected guidelines. There are several possible outcomes/results of this search:
      • a)
        an answer to PICO (a recommendation) is detected there and is of high quality. This recommendation is adapted.
      • b)
        an answer to PICO does not appear in existing CPG so there is nothing to adapt. The lack of recommendation should be highlighted in the final report.
      • c)
        the recommendation answering the PICO is of poor quality due to - Outdated literature (it can be detected by chance if a task force member knows relevant new evidence not included in the original CPG) and/or - lack of recommendation strength. The task force will decide if this recommendation can be used with some explanation or should not be used.
      • d)
        there are differences in recommendations across various guidelines and no consensus recommendation can be made (see 2.7 below).

      2.7 Ways to manage any disagreements between recommendations from CPG of the same quality

      In case of discordant recommendations the task force should review the evidence (provided by the original CPG authors) supporting these recommendations and decide whether to:
      • a)
        select one of them
      • b)
        include all with explanation for the disagreement particularly if all recommendations are directly related to Internal medicine patients (see 2.8 below).
      In most cases the task force is not expected to perform a new literature search to refine the recommendation.

      2.8 Focus of recommendations on the internal medicine patients (e.g. age, frailty and multimorbidity appropriately taken into account)

      Patients of advanced age or who are frail with multiple comorbidities are frequently managed in internal medicine departments. This population may impact recommendations and their applicability. In case there are no relevant recommendations in the original CPGs, the panel should answer the following questions:
      • a)
        Was the problem of multiple comorbidities considered in the text of original CPG?
      • b)
        What are the implications of multiple comorbidities and older age in the specific clinical situation (the topic of CPG)?
      • c)
        Should any good practice statements considering these specific populations be formulated?
      • d)
        Were drug adverse effects and drug interactions considered in the original CPG?
      Given the gap that exists between the reality of the care of patients with multiple comorbidities and the recommendations from CPG focused on a single disease, a priority should be to check whether there are sections in the CPG that address the impact of multiple comorbidities and pharmacotherapy on prevention, diagnosis and management recommendations. Even if the problem was not considered in the original guidelines, many of the original recommendations could apply indirectly to internal medicine patients. The task force should remark on this issue in their report.
      When referring to a single pathology CPG:
      • a)
        The task force should consider the most frequently occurring combination of comorbidities that coincide with this single disease (i.e., with cardiac diseases: diabetes, COPD, obesity or renal failure; with COPD: tobacco dependence, alcohol abuse or arteriosclerosis).
      • b)
        The task force should assess whether the guideline literature review includes original articles that take into account aged individuals with multiple comorbidities or using multiple medications (e.g.: higher risk of hypoglycemia in older patients, patients with renal disease using beta-blockers). If such original articles are included in the CPG the task force could either deliver a summary of them or only mention that there are original studies addressing these two issues. The latter option is preferable.
      • c)
        As some PICOs can be answered only if drug side effects and interactions are appropriately considered, the task force should assess whether the original CPG literature review includes articles that discuss drug side effects and interactions.
      Sometimes it may be necessary to refer to CPG related to other subspecialties than those directly related to CPG topic (e.g. while answering the PICO – Should macrolides be used in the presence of cardiac arrhythmia in patients with community acquired pneumonia?– referring to both pulmonology and cardiology CPG may be needed).

      2.9 Editorial process and dissemination

      2.9.1 Structure of the report

      The report from the task force should have no more than 3000 words (excluding references). It should contain:
      • a)
        a brief introduction highlighting the rationale for the article
      • b)
        a brief mention of the CPG considered for adaptation (rejected and accepted).
      • c)
        all necessary explanations for the discrepancy between guidelines and/or recommendation limitations
      • d)
        a list of PICOs and related recommendations (with the strength and quality of evidence from each of the CPG selected)
      • e)
        a short summary of the answers to the PICO questions accompanied, when possible, with a flow chart or clinical pathway
      • f)
        a short discussion of the issues related to internal medicine practice, emphasizing the aspects needed for better knowledge and research
      • g)
        if possible a short discussion with some remarks on implementation, but as healthcare provision is varied across the EFIM member countries a single suggested method of implementation is unlikely to be useful
      • h)
        references (see 2.9.3 below).

      2.9.2 External validation

      Before submitting the report for publication, it will be reviewed by the EFIM CPG-WG.
      External review will be done by:
      • a)
        EFIM Executive Committee
      • b)
        the editor of the European Journal of Internal Medicine.

      2.9.3 References

      The CPG which were evaluated, selected or rejected should be referenced. The main references supporting the recommendations mentioned in the report should also be included, as well as references to new evidence on potentially outdated recommendations.

      2.9.4 Supportive tools

      A check list for the task force members and external reviewers is provided (see Supplementary material).

      2.10 Management of the potential conflicts of interest

      Every CPG task force member should reveal any potential conflict of interest. An EFIM Executive Committee will decide if there is a risk that the conflict of interest of any member may compromise the reliability of CPG adaptation. In such case this particular member will be excluded from the task force.

      3. Discussion

      The article describes the methodology which will be used to adapt the published valid CPG to everyday internal medicine practice in Europe. A pragmatic, reliable, scientific and reproducible framework which allows a quick and smooth adaptation process is presented. Furthermore, the described methodology facilitates the dissemination and implementation of evidence based recommendations. There are many problems related to the use of guidelines, some of the most significant are: 1) the noise produced by the overabundance of CPG on the same topic, frequently with contradictory recommendations [
      • Li Q
      • Li X
      • Wang J
      • Liu H
      • Kwong JS
      • Chen H
      • et al.
      Diagnosis and treatment for hyperuricemia and gout: a systematic review of clinical practice guidelines and consensus statements.
      ]; 2) CPG with unspecific recommendations that do not apply to many patients in internal medicine practice [
      • Ruiz-Ruiz F
      • Medrano FJ
      • Navarro-Puerto MA
      • Rodríguez-Torres P
      • Romero-Alonso A
      • Santos-Lozano JM
      • et al.
      Delphi-RAND consensus of the Spanish society of internal medicine on the controversies in anticoagulant therapy and prophylaxis in medical diseases.
      ], and 3) the challenge to implement recommendations to patients not well represented in the trials which are the basis for the CPG recommendations, for example to elderly patients or people with many comorbidities [
      • Greenfield S
      • Billimek J
      • Pellegrini F
      • Franciosi M
      • De Berardis G
      • Nicolucci A
      • et al.
      Comorbidity affects the relationship between glycemic control and cardiovascular outcomes in diabetes: a cohort study.
      ].
      The strength of the suggested adaptation approach is the ability to cope with the issues mentioned above. The EFIM CPG-WG decided to use valid existing CPG instead of developing new documents. It is attempted to stimulate the use of CPG by selecting and refining the recommendations which are most likely to meet patients’ needs, and facilitate the implementation of CPG and provide a homogeneous European standard of practice.
      The method proposed was developed with a structured consensus, which was mainly based on two best tools for guideline quality assessment (AGREE II) [] and guideline adaptation (ADAPTE) []. Well accepted guidance proposed by other scientific societies was also followed [
      • Fabbri LM
      • Boyd C
      • Boschetto P
      • Rabe KF
      • Buist AS
      • Yawn B
      • et al.
      How to Integrate multiple comorbidities in guideline development: article 10 in integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report.
      ,
      • Uhlig K
      • Leff B
      • Kent D
      • Dy S
      • Brunnhuber K
      • Burgers JS
      • et al.
      A framework for crafting clinical practice guidelines that are relevant to the care and management of people with multimorbidity.
      ].
      Additionally, the PICO lists outlined in this adaptation process will facilitate the assessment of currently available evidence informing care of individuals with multimorbidity or aged [
      • Sinnige J
      • Braspenning J
      • Schellevis F
      • Stirbu-Wagner I
      • Westert G
      • Korevaar J
      The prevalence of disease clusters in older adults with multiple chronic diseases–a systematic literature review.
      ], and could be the source of practical research questions to be considered in future clinical trials and by CPG developers [
      • Boyd CM
      • Darer J
      • Boult C
      • Fried LP
      • Boult L
      • Wu AW
      Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: implications for pay for performance.
      ]. The series of CPG summaries for internists that will be prepared for publication with the usage of the methodology described in this article can make a great contribution to appropriate assessment of patient values and preferences in the context of multimorbidity and can help in weighting treatment benefits and harms. Such CPG summaries can inform decision-making for example in patients with multimorbidity. Some limitations of the suggested adaptation process should be noted. Firstly, there is a limitation that the process does not encompass gathering of new evidence neither its secondary synthesis. The effort is focused on pragmatic selection of the available evidence. Secondly, although the process is clearly defined with all necessary detail, the way of preparing the subsequent CPG summaries may vary depending on the task force and will need a constant monitoring by the CPG-WG. Finally, the described methodology will require validation. Its usefulness for CPG implementation and the consistency of the documents will need to be evaluated. As the new CPG summaries are prepared, probably further refinement of the methods will be required.

      4. Conclusion

      The EFIM CPG adaptation process is a tool that facilitate the use of valid CPG and implementation of recommendations by adapting general recommendations to specific clinical situations (elderly patients with comorbidities) that the European internists face in their day-to-day practice.

      Declaration of Competing Interests

      The authors nominated and all members of the Critical appraisal Guidelines Working Group CPG reveal that they have no potential conflict of interest related to the elaboration of this manuscript. Nor conflicts related with pharma industries or editorial or other trust, neither with EFIM Executive Committee which proposed the elaboration of the manuscript.

      Acknowledgment

      OTHERS MEMBERS of the EFIM Critical Appraisal of Guidelines Working Group that contribute to the paper are: Araújo F; Becattini C; Bergman J; Bevc S; Biscup E; Bojunga J; Kokorin V; Lander M; Lovic D; Manolev A; Marra A; Montano N; Unal S; Vartdal T; Yıldız P.

      Appendix. Supplementary materials

      References

        • Seto K
        • Matsumoto K
        • Fujita S
        • Kitazawa T
        • Amin R
        • Hatakeyama Y
        • et al.
        Quality assessment of clinical practice guidelines using the AGREE instrument in Japan: a time trend analysis.
        Plos One. 2019 May 2; 14 (eCollection 2019)e0216346https://doi.org/10.1371/journal.pone.0216346
        • Zhou H
        • Zhang S
        • Sun X
        • Yang D
        • Zhuang X
        • Guo Y
        • et al.
        Lipid management for coronary heart disease patients: an appraisal of updated international guidelines applying appraisal of guidelines for research and evaluation II-clinical practice guideline appraisal for lipid management in coronary heart disease.
        J Thorac Dis. 2019 Aug; 11: 3534-3546https://doi.org/10.21037/jtd.2019.07.71
        • The GRADE Working Group
        Grading quality of evidence and strength of recommendations.
        BMJ. 2004; 328: 1490
        • Guyatt G
        • Oxman AD
        • Akl EA
        • Kunz R
        • Vist G
        • Brozek J
        • et al.
        GRADE guidelines: 1. Introduction: GRADE evidence profiles and summary of findings tables.
        J Clin Epidemiol. 2011; 64: 383-394
        • Alexander PE
        • Brito JP
        • Neumann I
        • Gionfriddo MR
        • Bero L
        • Djulbegovic B
        • et al.
        World Health Organization strong recommendations based on low-quality evidence (study quality) are frequent and often inconsistent with GRADE guidance.
        J Clin Epidemiol. 2016 Apr; 72: 98-106https://doi.org/10.1016/j.jclinepi.2014.10.011
        • Li Q
        • Li X
        • Wang J
        • Liu H
        • Kwong JS
        • Chen H
        • et al.
        Diagnosis and treatment for hyperuricemia and gout: a systematic review of clinical practice guidelines and consensus statements.
        BMJ Open. 2019 Aug 24; 9e026677https://doi.org/10.1136/bmjopen-2018-026677
        • Jansen J
        • McKinn S
        • Bonner C
        • Les Irwig
        • Doust J
        • Glasziou P
        • et al.
        Systematic review of clinical practice guidelines recommendations about primary cardiovascular disease prevention for older adults.
        BMC Fam Pract. 2015 Aug 20; 16: 104https://doi.org/10.1186/s12875-015-0310-1
        • Marengoni A
        • Winblad B
        • Karp A
        • Fratiglioni L
        Prevalence of chronic diseases and multimorbidity among the elderly population in Sweden.
        Am J Public Health. 2008 Jul; 98: 1198-1200https://doi.org/10.2105/AJPH.2007.121137
        • Barnett K
        • Mercer SW
        • Norbury M
        • Watt G
        • Wyke S
        • Guthrie B
        Epidemiology of multimorbidity and implications for health care, research, and medical education: a crosssectional study.
        Lancet. 2012; 380: 37-43https://doi.org/10.1016/S0140-6736(12)60240-2
        • Mangin D
        • Bahat G
        • Golomb BA
        • Mallery LH
        • Moorhouse P
        • Onder G
        • et al.
        International group for reducing inappropriate medication use & polypharmacy (IGRIMUP): position statement and 10 recommendations for action.
        Drugs Aging. 2018 Jul; 35: 575-587https://doi.org/10.1007/s40266-018-0554-2
        • O'Mahony D
        • Curtin D.
        Deprescribing in older people: why it matters in routine clinical practice.
        Pol Arch Intern Med. 2018 Apr 30; 128: 197-199https://doi.org/10.20452/pamw.4255.Epub 2018 Apr 27
        • Van de Werf F.
        Reperfusion treatment in acute myocardial infarction in elderly patients.
        Kardiol Pol. 2018; 76: 830-837https://doi.org/10.5603/KP.a2018.0092.Epub 2018 Apr 10
        • Ena J
        • Gómez-Huelgas R
        • Romero-Sánchez M
        • Gaviria AZ
        • Calzada-Valle A
        • Varela-Aguilar JM
        • et al.
        Diabetes and obesity working group of the spanish society of internal medicine. Hyperglycemia management in patients admitted to internal medicine in Spain: a point-prevalence survey examining adequacy of glycemic control and guideline adherence.
        EJIM. 2015 Jul; 26: 392-398https://doi.org/10.1016/j.ejim.2015.04.020
        • Appraisal of Guidelines, Research, and Evaluation in Europe (AGREE) Collaborative Group
        For the AGREE collaborative group. Guideline development in Europe: an international comparison.
        Int J Technol Assess Health Care. 2000; 16: 1036-1046
      1. https://www.agreetrust.org/wp-content/uploads/2017/12/AGREE-II-Users-Manual-and-23-item-Instrument-2009-Update-2017.pdf.

      2. https://g-i-n.net/document-store/working-groupsdocuments/adaptation/adapte-resource-toolkit-guideline-adaptation-2-0.pdf.

        • Ruiz-Ruiz F
        • Medrano FJ
        • Navarro-Puerto MA
        • Rodríguez-Torres P
        • Romero-Alonso A
        • Santos-Lozano JM
        • et al.
        Delphi-RAND consensus of the Spanish society of internal medicine on the controversies in anticoagulant therapy and prophylaxis in medical diseases.
        Intrombin Project Rev Clin Esp. 2018; 218: 358-371https://doi.org/10.1016/j.rce.2018.04.006
        • Greenfield S
        • Billimek J
        • Pellegrini F
        • Franciosi M
        • De Berardis G
        • Nicolucci A
        • et al.
        Comorbidity affects the relationship between glycemic control and cardiovascular outcomes in diabetes: a cohort study.
        Ann Intern Med. 2009; 151: 854-860
        • Fabbri LM
        • Boyd C
        • Boschetto P
        • Rabe KF
        • Buist AS
        • Yawn B
        • et al.
        How to Integrate multiple comorbidities in guideline development: article 10 in integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report.
        Proc Am Thorac Soc. 2012; 9: 274-281
        • Uhlig K
        • Leff B
        • Kent D
        • Dy S
        • Brunnhuber K
        • Burgers JS
        • et al.
        A framework for crafting clinical practice guidelines that are relevant to the care and management of people with multimorbidity.
        J Gen Intern Med. 2014; 29: 670-679
        • Sinnige J
        • Braspenning J
        • Schellevis F
        • Stirbu-Wagner I
        • Westert G
        • Korevaar J
        The prevalence of disease clusters in older adults with multiple chronic diseases–a systematic literature review.
        PloS ONE. 2013; 8: e79641
        • Boyd CM
        • Darer J
        • Boult C
        • Fried LP
        • Boult L
        • Wu AW
        Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: implications for pay for performance.
        JAMA. 2005; 294: 716-724