This project was funded by the National Institute for Health Research Health Technology Assessment (NIHR HTA) Programme (project number 05/52/01) and was published in full in Health Technology Assessment 2009; Vol.13: No.59

The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the HTA programme, NIHR, NHS or the Department of Health.

Below is a list of relevance, reliability and transferability questions to ask when considering the adaptation of information and/or data on economic evaluations (box 10).

Question Box 10: Economic evaluation questions

To assess relevance and reliability7

1. Was a well-defined economic question posed in an answerable form?

2. a) What is the question being asked in the report?

b) Is the economic question relevant?

c) What type of economic analysis is being performed to answer the question (i.e. cost-minimisation, cost consequences analysis, cost-effectiveness analysis, cost-utility analysis, cost-benefit analysis)?

3. a) Has the viewpoint or perspective for the analysis been stated clearly, along with the reasons for this choice?

b) Is it a societal perspective, third-party payer perspective, or patient perspective?

c) Is the analysis presented in a disaggregated fashion showing these perspectives separately?

4. Was a comprehensive description of the competing alternatives given (i.e. can you tell who did what to whom, where and how often)?

5. Has the study included a comparison of alternative treatments for patients with the same clinical condition? Are those alternatives explicitly stated? Are the alternatives chosen valid and reasonable?

6. a) Has the evidence of the product‟s efficacy been established through randomised trials?

b) Has the evidence of efficacy been supplemented by evidence of effectiveness applicable to the patient population or subgroups considered in the study?

c) Has the latter evidence been derived from studies documenting routine use in clinical practice?

d) Have all the relevant and significant variations in effectiveness for different subgroups been identified and reported?

7. Was the effectiveness of the programmes or services established?

8. a) Are the methods and analysis displayed in a clear and transparent manner?

b) Are the components of the numerator (cost of each alternative) and denominator (clinical outcomes of each alternative) displayed?

c) Are clinical outcomes expressed first in natural units and then translated into alternative units, such as benefits or utility?

9. Are all important and relevant costs and consequences (outcomes), including adverse effects for each alternative identified?

10. Were costs and consequences measured accurately in appropriate physical units (e.g. hours of nursing time, number of clinician visits, lost work-days, gained life-years)?

11. How is Health Related Quality Of Life (HRQOL) measured?

12. a) Is HRQOL an important component of an economic analysis for this question?

b) Based on the sensitivity analysis how sensitive is the estimate of cost-utility to variations in HRQOL?

13. Were costs and consequences valued credibly?

14. Were costs and consequences adjusted for differential timing?

15. Are costs and consequences modelled (as a decision trees) with information derived from a variety of sources or estimated directly from specific patient population(s)?

16. a) Are capital costs and overhead costs included as well as operating costs? b) How are they measured?

17. How have indirect costs (i.e. productivity costs, cost of lost time) been identified and estimated?

18. a) For variables which are difficult to measure, what method is used to handle this difficulty?

b) Does this method slant the analysis all in favour of one intervention in order to bias the analysis against the expected result?

19. Was an incremental analysis of costs and consequences of alternatives performed?

20. Was allowance made for uncertainty in the estimates of costs and consequences?

21. Were adequate sensitivity analyses undertaken i.e. when parameters with high uncertainty were analysed, did the direction of the results change?

22. If a stochastic sensitivity analysis was applied, are the underlying distribution functions justified?

23. What equity assumptions have been made in the analysis? For example, are QALYs gained by any individual considered equal?

24. a) Is the incremental cost-effectiveness ratio estimated for a specific clinical indication that represents the majority of all of its expected use by those covered under the programs operated by the decision-makers to whom the report is addressed?

b) Are there other indications which have not been considered which involve a large amount of utilization for which the ratio may be very different?

25. a) Is there an estimate of the aggregate incremental expenditure required for the decision-makers to whom the study is addressed, to provide this product to patients covered by their programs?

b) What is the estimate of aggregate incremental costs?

c) Does this estimate cover all of the major indications for use of the product?

26. Did the presentation and discussion of study results include all issues of concern to users?

To assess transferability8

27. How generalisable and relevant are the results, and validity of the data and model to the relevant jurisdictions and populations?

28. a) Are there any differences in the following parameters?

I. Perspective

II. Preferences

III. Relative costs

IV. Indirect costs

V. Discount rate

VI. Technological context

VII. Personnel characteristics

VIII. Epidemiological context (including genetic variants)

IX. Factors which influence incidence and prevalence

X. Demographic context

XI. Life expectancy

XII. Reproduction

XIII. Pre- and post intervention care

XIV. Integration of technology in health care system

XV. Incentives

b) If differences exist, how likely is it that each factor would impact the results? In which direction? Of what magnitude?

c) Taken together, how would they impact the results and of what magnitude?

d) Given these potential differences, how would the conclusions likely change in the target setting? Are you able to quantify this in any manner?

29. Does the economic evaluation violate your national/regional guidelines for health economic evaluation?

Answers to these questions should help the user extract information and/or data from this section of the HTA report. This „adaptation material‟ on economic evaluation can be incorporated within an HTA report in your own setting. There may be a need to update these data and supplement it with local context data.

Jurisdiction is the authority given to a legal body, or to a political leader (Prime Minister, President, etc.) to deal with legal matters, and to pronounce or enforce legal matters. Jurisdictions are the territorial areas (eg. countries or regions) where particular laws or guidance (including policy decisions) apply.

7 Questions taken from CCOHTA Guidelines for economic evaluation of pharmaceuticals, 2nd edition: Canada. Ottawa: Canadian Coordinating Office for Health Technology Assessment (CCOHTA); 1997 and Drummond MF, O'Brien BJ, Stoddart GL, and Torrance GW. Methods for the Economic Evaluation of Health Care Programmes. Oxford Medical Publications, UK, 1997; 2. edition.  

8 List of factors taken from Welte, R. und Leidl, R., 1999, Übertragung der Ergebnisse ökonomischer Evaluationsstudien aus dem Ausland auf Deutschland: Probleme und Lösungsansätze, in: Leidl, R., Graf von der Schulenburg, J. M. und Wasem, J. (Hg.): Ansätze und Methoden der ökonomischen Evaluation. Eine internationale Perspektive, Baden-Baden: Nomos.