Kent Fire and Rescue Service
Evaluation Framework 2021

KFRS uses both qualitative and quantitative study designs, and their associated methods to evaluate intervention activities. One single method is rarely suitable to offer robust evaluation, and often many methods are utilised in any given evaluation to gain depth and breadth of information about the impact of the intervention activity. The evaluation method chosen needs to be suitable for the intervention activity and target audience. As detailed above, it should also be proportionate to the anticipated impact and cost of the intervention. This means that evaluation methods are chosen based on each intervention that is being evaluated, the type of evaluation needed, and the likely impact of the intervention. 

Below details some of the methods KFRS have used or are likely to use based on the types of intervention currently undertaken. Where possible small-scale pilot studies are conducted to test both the intervention content and evaluation methods before a wider roll-out. This may mean completing a shorter PDSA cycle before moving on to the main study in the overall cycle for that intervention activity. 

When selecting a specific evaluation methodology, the NSMC considered what is identified as ‘best practice’ as per the Hierarchy of Evidence (Figure 3), whilst also considering what is realistic to conduct within KFRS’s capacity. The quality of the evidence is determined by the methods used to minimise bias within a study design. Randomised controlled trials (RCTs), give the highest level of evidence for certain types of research; however they are also costly to deliver and resource intensive. It can be difficult to gain enough control in real-world settings to conduct a fully randomised control trial, however in collaboration with partners such robust methods may be achievable. 

Figure 3: Hierarchy of evidence

Based on the hierarchy of evidence (Figure 3), a series of recommended methods and/or study designs can be linked to different areas of intervention activity. These may vary in practice when more is known about the specific research questions and depth of intervention required for each intervention. Broadly speaking, however, it is suggested that for the education programme pre- and post-intervention non-randomised designs are used, Safe & Well/Home fire safety visits are evaluated post-intervention only, although with the resources in place, a cluster randomised control trial (with waitlist control) may be possible. For public safety campaigns and events a lighter touch approach is required due to the limited control over delivery of intervention content, and so generally a post-intervention design would be suggested. Buildings safety intervention would be suitable for quasi-experimental pre- and post-intervention design, or an experimental approach to testing message efficacy with different types of building/business owner. 

Table 2: Study designs and evaluation methods

Below we detail the evaluation methods that could be used to evaluate KFRS intervention activity. It is important to highlight that the evaluation methodology and study design should be appropriate to the research question we are seeking to answer, relevant to the intervention activity, linked to the relevant behaviour change theory, and importantly, proportionate to the impact and cost that the intervention activity is likely to have. When deciding which methods to use, we must also consider the challenges we face in evaluating KFRS’s fire safety intervention activity (see above). As one of the first UK FRS’s to commit to research and evaluation in-house, KFRS aims to conduct robust research and evaluation that can be used by other FRS’s nationally as best practice. Furthermore, it is likely that multiple methods may be utilised to evaluate some interventions to gain a holistic view of the impact generated from the intervention.