Planning an evaluation of telecare and telehealth
Professor Stan Newman and Anna Davies, University College London
This article was prepared by Mike Clark, co-project lead for WSDAN, based on the presentation made by Professor Stan Newman to the WSDAN Roadshow in Manchester on 10 December 2009. The presentation can be downloaded from our past events pages.
Professor Stan Newman, supported by Anna Davies, provided an important presentation on evaluation of telecare and telehealth, which will be of interest to many local authorities and primary care trusts (PCTs) looking at potential outcomes from their own programmes.
Professor Newman explained what makes a good-quality evaluation, and presented some of the different methods that can be used. He also explored some of the practical issues involved in conducting an evaluation.
What makes a good evaluation?
Evaluation is the systematic acquisition and assessment of information or data about something, and its purpose is to inform decision-making. The Whole System Demonstrator (WSD) programme is being carried out to provide evidence to support an evidence-based care approach – that is, the use of telecare and telehealth within an integrated health and social care setting. Local authorities and PCTs are interested in carrying out their own evaluations to support their commissioning and implementation programmes.
An evaluation can be used to answer questions about the clinical effectiveness and cost-effectiveness of telecare and telehealth. It can also be used to examine the organisational changes that would need to be made to provide an effective service. Most importantly, it can assist with decisions about whether remote monitoring is the right service for patients, users and carers.
A good evaluation should aim to produce good-quality information to answer the question or questions posed and provide a true reflection of what is happening in telecare and telehealth in the area studied.
A hierarchy of evidence
Professor Newman presented a hierarchy of evidence, from expert opinion through to case studies or reports, uncontrolled observational studies (eg, ‘before and after’ studies), controlled observational studies, randomised controlled trials, and systematic reviews and meta-analysis. Much of the available evidence for telecare and telehealth lies at the bottom of the hierarchy – that is, expert opinion and case studies. The WSD programme was under way to tackle the lack of robust, good-quality evidence.
Methods of evaluation
In his presentation, Professor Newman selected two areas from the hierarchy of evidence.
First, the randomised controlled trial (RCT), where people are allocated at random to intervention and control groups. The method used in the WSD trial is clustering, where patients are allocated according to their GP practice. The control group receives normal care compared to the intervention group, which receives the technology. The control group minimises selection bias (such as providing technology to those more likely to want it) and reduces other potential explanations for outcomes. An RCT is the most rigorous form of trial, and the greater the number of people in the trial, the greater the statistical significance of the outcomes. The WSD programme has a number of independent evaluators covering five key themes.
Second, Professor Newman selected a ‘before and after’ trial, where measurements are made before an intervention and again after the intervention. This type of trial avoids the complexities of an RCT. However, it is not possible to rule out other explanations for any differences seen even if there is a control group.
He also suggested some other approaches – for instance, using similar criteria and measurements to the WSD programme and linking to the WSD control group data where the separate trial does not have a control group.
There is also value in carrying out qualitative evaluations, which can provide rich descriptive information with answers to questions such as ‘Is telecare or telehealth acceptable to clients/patients and carers?’ or ‘What do service providers think about this approach?’ A qualitative approach involves appropriate sampling and interview methods with clear analytic methods, ethical considerations, analysis and a clear link between the data collected and conclusions.
Practical issues involved in conducting an evaluation
Organisations planning to undertake an evaluation need to consider a number of practical issues, including the resources required to complete the study. Evaluations need to involve all relevant staff as early as possible in the process, and consider the impact on work patterns, job roles and workloads.
There are no set criteria for how long an evaluation should take. It can depend on the evaluation question or questions. For instance, a longer evaluation would be needed to detect physiological changes, whereas a shorter period may be sufficient if the question is around initial acceptance of the technology by an individual. At present, we know very little about the longer-term deployment of telecare and telehealth equipment.
In some circumstances, approval may be needed from other departments (ethics, and research and development (R&D)), for example, where a new treatment is being compared with normal care or where there is collection of data other than normal NHS data.
The WSD programme uses 24 questionnaire instruments that are validated and reliable. Additionally, specific measures can be used to answer evaluation questions. Examples of questionnaire instruments used include patient/client outcomes such as quality of life, diabetes health profile, HbA1c and carer stress.
Conclusion
Professor Newman considered that randomised controlled trials offer the best quality of evidence about effectiveness of telecare and telehealth, but alternative methods are available (eg, using control group data from the WSD programme in a ‘before and after’ study). Qualitative evaluations can be useful to provide information to answer different questions. Sufficient consideration should be given to planning an evaluation.