Kent Fire and Rescue Service
Evaluation of Safe & Well Visits 2019/20

In 2018 Kent Fire & Rescue Service (KFRS) commissioned the National Social Marketing Centre (NSMC) to develop an evaluation framework around its customer safety interventions. The aim of this was to provide guidance on methods and data collection tools which could be implemented to determine the effectiveness of the interventions, and identify areas for improvements (NSMC, 2018). Based on the logic model (see section 6.1), which was designed around all customer safety activity, the current evaluation of Safe & Well visits considers short term outcomes (e.g. recall of information, change in knowledge/awareness, equipment fitted, and actions taken), some medium term outcomes (e.g. applying knowledge to behaviour, intention to change behaviour), and to some degree the long term outcome of self-reported behaviour change. In line with the NSMC recommendation, the evaluation used a cross-sectional post-intervention survey, using primarily qualitative, open-ended questions to gain depth of information, and allowed participants to offer explanation for their responses. In addition we sought feedback from staff who deliver Safe & Well (S&W) visits, via focus groups, to combine with customer feedback to give a more rounded view of the impact S&W visits have on customer safety. Responses were analysed using thematic content analysis. 

Note that at the time of this evaluation all home visits undertaken by KFRS are referred to as Safe & Well visits. This includes visits undertake by operational crews and a team of specialists.

1.1. Safe & Well visits in context

According to Home Office statistics, the number of Accidental Dwelling Fire (ADF) attended by Fire & Rescue Services (FRS) nationally has been relatively stable since 2012/13 (Home Office, 2019). In Kent and Medway, the number of ADF’s attended in 2018/19 totalled 523, including eight fatalities and 167 casualties. For April 2019 to February 2020, ADFs attended were 430, including four fatalities and 112 casualties. This highlights the need for customer safety interventions that encourage behaviours associated with fire safety and reducing risk of fires. Since 2004, promoting safety has been part of the statutory function of Fire & Rescue Services (FRS), and there is an expectation that one avenue to achieve this is to deliver S&W visits (home safety visits). The primary aim of S&W visits within Kent is the prevention of fatalities and casualties in ADFs, with the wider benefit of improving safety and wellbeing. 

Safe & Well visits in Kent are currently offered to households that include anyone aged over 70 years, anyone aged under five years (including pregnant women), smokers, or people with long-term health conditions. S&W visits are one of a number of customer safety and prevention-based intervention activities offered by Kent Fire and Rescue Service (KFRS), including educational school-based programmes, community-based safety events, and road safety education. In the most recent financial year (2019/20), the S&W team have completed 11,234 and stations 9,060 (provisional figures). Whilst this shows that a large number of visits are being completed, it does not evidence whether there is any behavioural impact of the current S&W provision. That is, the quantity of visits might be high, but there has until now been little evidence of quality of content (in relation to behaviour change and increased safety).

The quality of evidence evaluating home fire safety interventions has been found to be very low (Senthilkimaran, Nazari, MacDermid, Roche, & Sopko, 2019), meaning that despite a general trend suggesting that home fire safety interventions improve knowledge and behaviour, we cannot be confident that reductions in ADF’s is attributable to the intervention, or generalizable across different types of intervention. A major limitation with a number of previous studies is the assumption that increased quantity of intervention and observed decrease in number of fires are causally linked (Similar pitfalls are evident in other research such as the evaluation of the Fire Kills campaign in 2009 which identified changes in sales of smoke alarms, and the occurrence of smoke alarms in ADF incidents, assuming that changes were a direct result of media campaigning (Evans & Wright, 2009), further highlighting the need to consider customer behaviour in evaluation research). Furthermore, the content of evaluation varies and is not always focused on whether the intervention changes safety behaviours. A recent NFCC survey found that only nine of the 14 FRS’s that responded were using any form of evaluation of S&W visits, and of those most were either focused on quality assurance or were non-specific about how the evaluation is conducted (personal communication with Joanne Mann at Humberside FRS). This makes it difficult to judge how effective S&W visits are in changing behaviour to encourage safer actions. 

1.2. Evidence from other Safe and Well evaluations

S&W visits in the UK have developed over many years since around 1999 and promoting fire safety became a statutory function as part of the 2004 Fire and Rescue Service Act. Home Fire Safety Checks were partially funded by the government from 2004 to 2008. Since then each fire & rescue service has developed their own visits with little guidance, although the National Principles for Safe & Well were published in 2015 and new guidance for a person-centred home fire safety visit is currently being developed by the NFCC. As a result the core fire safety content of S&W visits across UK FRSs is broadly similar and at least more similar than to S&W visits (or equivalent (This paper refers to all visits as S&W since this is the current (at the time) approach taken by KFRS. All visits to a customer’s home are referred to as S&W visits, although other FRS’s differentiate S&W from Home Fire Safety Check/Visits. For consistency we refer to all visits conducted by FRS personnel at the customers’ home as S&W throughout the paper)) in other countries. 

Therefore, this review focuses on research from the UK and in particular, research that aims to evaluate the effectiveness of S&W visits by FRS’s. A systematic review of interventions was conducted which included studies of any fire safety intervention globally (Senthilkumaran et al., 2019). Only studies including an intervention evaluation with a comparison group (other intervention or control) were included, leaving a total of ten papers. Of those only three were conducted in the UK, and of those only one with adults (or not a schools programme). This single study evaluated an injury prevention briefing, and therefore may not be typical of home fire safety visits as defined in other research papers. Overall, the review and meta-analysis identified that the little evidence that does exist is low quality and shows little to no impact of the identified interventions. However, the interventions all differed, and were conducted in different countries, making comparison very difficult. The authors also acknowledge the difficulty in accounting for fires that do not occur and how to assess the impact of intervention based on preventing something. With that in mind, they support a focus on knowledge and behaviours that might be indicative of future safety.

In the UK, evaluation of S&W visits is still developing and of the evaluation studies that were accessible for this report, each one differed in its approach. For example, Cheshire FRS compared CFRS data to a comparison group comprised of 37 other non-metropolitan FRS, counting the number of S&W visits per 100,000 population between 2001/02 and 2010/11 (Arch & Thurston, 2012). They used ‘number of ADF’s’ and ‘injuries resulting from ADF’s’ as a measure of success of the S&W visits, assuming that a reduction in these numbers indicated effective adoption of S&W advice. Similarly, Essex FRS evaluated a pilot intervention of S&W visits carried out by trained volunteers in 2016/17, comparing the number of ADF’s before and after the pilot intervention period (Yannitell & Chatsiou, 2019). Both studies concluded that the delivery of S&W visits was associated with fewer ADF’s, but assume that the number of S&W visits delivered is the driver of the fire activity. 

Some studies have considered the knowledge and behaviours of S&W customers following a visit. For example, Avon FRS interviewed 15 customers before, within two weeks, and within three months of their visit (Williams & Manning 2016). The majority of the people interviewed reported making changes, and sustaining those changes at three months post-visit. However, they identified that this was only the case for simple and easy to implement behaviours, and especially cost-free to the homeowner. Similarly, other research with Avon FRS identified particular reasons for not adopting safer behaviours included disruption to existing routines that could lead to distress (Bird, Tapp, Lancaster, & Clark, 2010). However, this was not directly evaluating S&W visits, but rather assessing attitudes towards fire safety among older adults. 

One of the more robust evaluations of S&W visits was conducted by Cordis Bright on behalf of LFB (London Fire and Emergency Planning Authority, 2013). The strength of their evaluation lies in the multifaceted approach, combining incident data, self-reported behaviour, and employee views. Analysis of authority data showed that in the six and a half years up to 2006, customers who received a S&W visit were ten times less likely than those 
who did not receive a visit to experience an ADF. The research also highlighted that homes who had received a S&W visits were more likely to become aware of the fire sooner (<5 minutes) and the fires were more likely to be rated as ‘slight’ or ‘moderate’ than ‘significant’ or ‘severe’ fires than where S&W visit were not conducted. The interviews with S&W customers found that only some had made changes to their behaviour, but most felt they had learnt new information. Similarly to Avon FRS’ study, LFB customers reported carrying out simple, cost-free behaviours, and reasons for not adopting any change were mostly due to perception of low risk and therefore not needed, or competing rationale such as not closing doors because that made them feel ‘hemmed in’.

KFRS have also previously looked at ways to gather feedback regarding S&W visits. In 2010 questions about the impact of S&W visits were included in public consultation, and showed that 40% of respondents reported having changed their behaviour as a result of safety messaging received via KFRS (Kent Fire & Rescue Service, 2010). Furthermore, research carried out as part of an MA by a KFRS employee sought feedback via postal survey from S&W customers (Stanford-Beale, 2010). Survey responses showed that 65% of customers reported changing their behaviour after the S&W visit, and the 32% who reported no behaviour change said that it was because they already followed the advice anyway. A quarter of the respondents reported that one change was the regular testing of smoke alarms, and 70% stated that they tested alarms more regularly after than before the visit. This seems promising, although other research suggests that smoke alarm testing and knowledge of smoke alarm status is often over-estimated (Chen, Gielen, & McDonald, 2003). Furthermore, within the KFRS study, open-ended responses seemed to sit at odds with the quantitative response that most people were checking their alarms. In the qualitative responses, only 18 respondents mentioned checking smoke alarms as an active behaviour. In fact, despite high proportions of quantitative responses suggesting behaviour change, there was far less evidence of this in the qualitative responses. This might support the notion that simply ticking a box encouraged responses in line with what might be expected, rather than a true reflection of behaviour. 

Although not directly evaluating S&W visits, some research has investigated knowledge and memory of a video-based fire safety intervention aimed at 12 older adults (Lehna, Merrell, Furmanek, & Twyman, 2017). This showed that fire safety knowledge was low at baseline (despite the majority of participants having had a S&W visit), but significantly increased at follow-up (T2 immediately after video) and retention (2 weeks later), and there was no significant difference between T2 and T3 (indicating that recall had not improved, nor had it fallen back below T2). This suggests that an educational video could also support the increase in, and retention of, knowledge relating to home fire safety. 

1.3. Rationale

Safe and Well visits are carried out by fire services across the country, however the effectiveness of these interventions is rarely reviewed. There are two ways of evaluating the effectiveness of the intervention; one is to look for changing patterns in incident data that may be attributed to S&W visits, and the other is to assess the impact at a customer level. To date, studies undertaken in this field have tended to focus on assessing the impact of visits based on changes in incident numbers. The challenge with this method is that it is difficult to directly attribute any change in incident numbers to S&W interventions due to the huge number of additional, unmeasured variables that may have affected the number of incidents at any given time. Therefore, for this study, it was decided that a more informative evaluation would measure customers’ self-reported changes in their behaviour resulting from the visit. To supplement this, we also used staff feedback via focus groups on perceptions of the impact of visits and likelihood of behaviour change.

Following the structure of a pre-defined evaluation framework (NSMC, 2018) we adopted a cross-sectional post-intervention evaluation of customer experiences during S&W visits. Qualitative data were collected through telephone interviews and were then thematically analysed, and trends in responses are reported below. To supplement this, we also conducted focus groups with staff who deliver S&W visits to allow triangulation of experiences. Outcome measures were based on the short, medium, and long term outcomes identified in the logic model (see section 6.1).