Community bushfire safety: a review of post-Black Saturday research

Abstract

Article

Introduction

Saturday 7 February 2009 brought predictions of the worst fire weather conditions in the Australian state of Victoria’s history. The weather conditions were significantly worse than predicted with a record high in the state capital’s CBD (Melbourne) of 46.5 degrees Celsius and higher temperatures elsewhere. The day came after a severe and protracted drought, with most of Victoria receiving below average rainfall in the previous 12 years and the area surrounding Melbourne receiving the lowest rainfall on record (Teague et al. 2009, p.36). 173 people lost their lives and more than 2000 homes were destroyed in the fires, as well as significant dollar losses and long-lasting intangible impacts on those affected. Approximately 6000 households were directly affected by the fires and thousands more were severely disrupted. In addition to the lives lost from the fires, health authorities estimate that the January heatwave contributed to the deaths of another 374 people (Department of Human Services 2009). The research reports reviewed in this paper are concerned only with bushfires.

A number of research reports on human behaviour and bushfire safety (referred to in this paper as ‘community safety’) have been presented to the Victorian Bushfires Royal Commission, including reports from the Country Fire Authority (CFA), Office of the Emergency Services Commissioner (OESC), Bushfire Cooperative Research Centre (CRC) and Department of Justice. These reports have different research aims and employ a range of methods and samples. Some reports investigate household safety issues during the Black Saturday bushfires, while others examine preparedness and intentions for future fires.

The OESC commissioned a study to review the post-fire research reports that were presented to the Commission. The aim of the study was: to identify common findings, inconsistencies and gaps relating to community bushfire safety; to document the implications of methodological differences for the research findings; and to suggest areas where further research and agency efforts are needed. This paper summarises aspects of that study. While the findings set out below are well-supported by data, it is important to recognise that generalised statements inevitably conceal the variation and complexity of individual and community attitudes, intentions and behaviours. Nevertheless, by examining and bringing together results from many reports through a systematic analysis, this study has produced well-grounded results. The detail of each report and of our analysis can be found in the full report prepared for the OESC (Whittaker and Handmer 2010).

Approach

Most reviews of individual studies are in narrative form. Among their shortcomings, narrative reviews typically do not discuss how studies were selected for inclusion (the sample of studies is typically based on the author’s whim, rather than on clear criteria), or provide explicit criteria for the assessment of the quality of, or the impacts identified by, different studies. This tends to hide any bias by the author(s), and importantly does not allow for findings to be replicated. In the medical and health sciences this has led to concern about the quality of reviews on which evidence based practice depends (Hemingway and Brereton 2009).

The idea of a systematic review evolved from these concerns with the intention of applying the same rigour in secondary research as is expected in primary research: “a systematic review is an overview of primary studies which contains an explicit statement of objectives, materials, and methods and has been conducted according to explicit and reproducible methodology” (Greenhalgh 1997). A meta-analysis is a statistical synthesis of the results of a number of primary studies that addressed the same question in a similar or ideally the same way. These approaches are now very widely used and have become routine in medical science (e.g. see the Cochrane Collaboration), and the techniques of systematic reviewing are developing rapidly (Hemingway and Brereton 2009).

Despite the apparent benefits, systematic reviews are subject to a number of important criticisms (Greenhalgh 1997; Hemingway and Brereton 2009). Many reviews do not follow the ideal review process. A major failing is often the selection of studies to include, with unpublished material often being ignored. Another issue is the combination of studies undertaken with different aims at different times and places (Eysenck 1995).

Here we do not undertake a full systematic review, but attempt to address some of the fundamentals of that approach, in particular the issues of study inclusion, quality, and interpretation of findings in the context of the individual study aims and methodology. The quality of research was assessed before inclusion, but this was not done using external blind peer reviewing.

All of the post-fire studies prepared for the Royal Commission were assessed as being of reasonable quality and are included, although the study on refuges of last resort is not included in most of the analysis due to its specific focus. It should be noted that none have been published in the scientific literature at this stage, but some were subject to review as part of internal quality control processes.

The research reports

For this study we reviewed the major community safety research reports presented to the Royal Commission, with the exception of a report dealing solely with fatalities (Handmer et al. 2010). These included reports from the CFA, the OESC, the Bushfire CRC and the Department of Justice.

Table 1 presents detail of the nine studies that were reviewed, including the aims, methods of data collection, sampling, timing and analysis of each, and their implications for research findings¹. Readers are referred to the full review (Whittaker and Handmer 2010) and the individual studies (listed at the end of this paper) for more detailed discussions of research methodologies and findings.

Table 1: Summary of the reports examined.

Research aims and methods

The aims of a research project shape results by guiding the questions researchers ask and the methods that are used to investigate them. Aims vary and are a key issue in comparison. A distinction can be drawn between the reports that aim to investigate general attitudes, intentions and behaviours related to bushfire risk, and those that aim to investigate aspects of a specific event or threat. The OESC, Department of Justice and first CFA reports all investigate general attitudes, intentions and behaviours to bushfire risk, with a focus on what people intend to do in various circumstances. In contrast, the second CFA report and both Bushfire CRC reports focus on actual behaviours and responses to specific events or threats, in addition to intentions. For example, the CFA report investigates responses to an actual declaration of a Code Red/catastrophic bushfire danger day, and both Bushfire CRC reports investigate both intended and actual responses to the 7 February bushfires.

While reports on general bushfire risk present findings on levels of awareness, preparedness and intended actions for future fires, they do not necessarily indicate how people will respond when confronted with a bushfire.

Questionnaires were the most common form of data collection. Questionnaires are a time and cost effective method of collecting large amounts of data that can be analysed relatively easily and quickly. The questionnaires used by the OESC, Department of Justice and CFA generated quantitative data. Those used by the Bushfire CRC research group contained both pre-coded responses and open-ended questions. These open-ended questions allowed respondents to explain their responses in their own terms and produced qualitative data. The Bushfire CRC research group’s first report was based on interviews that produced only qualitative data.

All methods of data collection contain inherent biases. For example, a landline phone survey is likely to exclude those who favour mobile phone use and/or do not have landlines – often younger people. Online surveys are likely to exclude those who do not have personal access to a computer and/or the internet, or who make limited use of the internet – often older people. There is potential for bias in any survey where a disproportionate number of people from an identifiable group do not respond.

All of the studies took measures to avoid sampling bias, such as taking a large, random sample and filling certain demographic and geographical quotas (Table 1). Findings may therefore be said to be more or less representative of the populations from which each sample was drawn. Studies by the Department of Justice, OESC and CFA variously sampled populations in the 52 high bushfire risk townships (as designated by the CFA), Fire Danger Rating (FDR) districts, and the state as a whole. Studies by the Bushfire CRC were principally concerned with community safety during the Black Saturday fires and therefore sampled populations within the impacted areas. These studies did not employ random sampling strategies due to the relatively small size of the impacted population and the need to collect as much data as possible. Clearly, people with recent experience of fire are more likely to be aware of fire-related issues and be more sensitive to safety concerns.

All of the studies were undertaken after the Black Saturday bushfires. Consequently, the data collected for each report is influenced by respondents’ recollections of the fires, subsequent media coverage, and heightened concern about the 2009/10 fire season. The semi-structured interviews undertaken by the Bushfire CRC immediately after the fires will have been influenced least by outside sources. Those conducted later in the year are likely to have been influenced by the considerable media coverage and new advice from agencies.

Review of key research findings

Analysis of the research reports identified a range of findings related to the following issues:

• Awareness and attitudes to bushfire risk;
• Planning and preparedness;
• Awareness and understanding of the Code Red Fire Danger Rating (FDR)²;
• Intended responses to Code Red declarations;
• Decisions about when to leave and where to go; and
• Intended responses for the 2008/09 and 2009/10 fire seasons;
• Actions taken during bushfires.

Readers are directed to the full report for a more detailed discussion of key research findings (Whittaker and Handmer 2010).

High level of awareness

The reports revealed a generally high level of bushfire awareness in high fire risk areas, with Department of Justice and Bushfire CRC research reporting that around 80% of respondents considered themselves at risk. This is consistent with CFA research that found high levels of acceptance of personal responsibility for bushfire safety, with more than 90% of respondents agreeing that: a bushfire may impact on their property; they must be self-sufficient in the event of a fire; and they are responsible for their home and property during the bushfire season.

However, interviews conducted immediately after the Black Saturday fires by the Bushfire CRC revealed that many people living in suburban locations had not considered themselves at risk prior to 7 February. This was confirmed in the CRC mail survey, where a greater proportion of respondents from fire affected parts of Bendigo (60%) and Horsham (49%) reported that they hadn’t considered or decided what they would do if a fire occurred, or had decided that they didn’t need to do anything.

Of course, findings related to bushfire awareness will have been influenced by recollections of the Black Saturday fires, subsequent media coverage and heightened concern about the 2009/10 fire season.

Variation in fire plans and preparedness

The reports reveal a great deal of variation in planning and preparedness. Reports by the Department of Justice, Bushfire CRC and CFA and all found that around two-thirds of households had fire plans, with around 20-25% having a written plan. However, the Bushfire CRC’s interviews with households affected by the fires revealed considerable variation in the quality of fire plans and their effectiveness during the fires. It also found that many people were not prepared for the severity of the February 7 fires. Around half of those surveyed by the Bushfire CRC rated their preparedness as high or very high; however, three-quarters wanted to be more prepared, suggesting that self-reported levels of awareness may be somewhat overstated.

Limited understanding of Code Red FDR

The research suggests a limited understanding of the new FDR system. CFA research found that around 60% of residents in high fire risk areas were aware that Code Red is the highest level of danger and that on Code Red days emergency services’ advice is to leave early. However, Department of Justice research found lower levels of understanding in high bushfire risk areas, with around one-fifth of respondents identifying Code Red as the highest FDR and one-third aware emergency services’ advice is to leave on these days.

There are a number of reasons why these findings are not strictly comparable. The CFA asked respondents if they knew what the highest fire danger rating is now called, providing them with a number of pre-coded response (e.g. ‘Code Red/catastrophic’; ‘Other description – extreme, severe, very high’). In contrast, the Department of Justice asked respondents to explain what the Code Red FDR means to them, recording responses verbatim. The findings presented in the report were generated by classifying responses into a much broader range of categories (e.g. ‘Evacuate/get out/get ready to leave’; ‘It’s the worst case/the highest rating’; ‘Put our fire plan into action’ etc.). All of these and other responses could conceivably be considered ‘correct’ given the question asked respondents what Code Red means to them. The Department of Justice research also considered closely-related responses such as ‘It’s the worst case/the highest rating’ and ‘It’s catastrophic’ separately. Had these responses been grouped together, the research would most likely have found that a higher proportion of people understood a Code Red warning to mean the highest or catastrophic fire danger. Reanalysis of the Department of Justice data could be useful.

Waiting

The reports consistently found that many people intend to wait for advice from emergency services or until they are directly threatened before taking action. CFA and OESC research found that around one-quarter of those who intend to leave would wait for advice from emergency services before leaving. Bushfire CRC research found that one-quarter of those affected by the Black Saturday fires intended to wait and see what the fires were like, or until they were directly threatened, before deciding to stay or go. This is supported by research into actual responses to the fires, which found that half of those who left considered themselves to have left ‘late’ or ‘very late’ and that one-third of those who stayed to defend left during the fire.

Variation in intended responses

The research reports present varied findings relating to people’s intended responses to bushfires. Research by the OESC and CFA found that during the 2008/09 bushfire season around two-thirds of those in high bushfire risk areas intended to leave their homes, while around 30% intended to stay. However, the Bushfire CRC’s survey of those directly affected by the Black Saturday fires found that half had intended to stay, with 19% reporting their intention to leave and 26% intending to ‘wait-and-see’. Similar differences can be seen in findings related to intentions for the 2009/10 fire season.

These differences reflect variation in the samples and timing of the research, as well as how questions were asked and data coded. For example, one OESC finding classified responses as ‘leave’ or ‘stay and defend’. The Bushfire CRC questionnaire included a much broader range of responses, including: ‘leave as soon as you know there is a fire threatening your town or suburb’; ‘wait to see what the bushfire is like before making a decision’; and ‘hadn’t thought about it’. Nevertheless, this does not explain the significantly higher proportion of Bushfire CRC respondents who intended to stay and defend in 2008/09. This discrepancy could reflect the influence on the OESC study of media coverage and new key messages that emphasise leaving on the people’s recollections of past intentions. Those who actually experienced the February 7th fires are more likely to have reflected on their real intentions prior to the fires and how these related to their actual response.

Research also suggests a gender dimension to intended responses, with women more likely to want to leave than men. This is supported by other recent bushfire research (Proudley 2008; Handmer et al. 2010; Haynes et al. 2010).

Major gap between intentions and actions

A number of the reports identify significant disparities between expressed intentions and actual responses to bushfire risk. Research by the Department of Justice, OESC and CFA found that 50-60% of residents intended to leave for Code Red days, with more than 60% intending to leave the night before or early in the morning. However, the CFA’s research following the declaration of a Code Red day in three FDR districts found that very few acted on their intention. It found that two-thirds of residents were at home on the day. Of the third that weren’t at home, just 1.5% left because it was a Code Red day. Furthermore, intentions appear to be influenced by perceived false alarms, with around three-quarters of residents indicating that they would not leave early on future Code Red days.

The Bushfire CRC’s research into responses to the Black Saturday fires found that a significantly higher proportion of respondents (around half) stayed to defend than is suggested by research into intended responses for the 2008-09 fire season. Highlighting the gap between intentions and actions, it found that half of those who intended to leave safely endangered themselves by leaving late, while a third of those who stayed to defend left at some stage during the fire. The study also highlights confusion about what it means to ‘leave early’ (see also Tibbits and Whittaker 2007).

Differences due to different methodologies

Most of the differences between the research reports are minor and can be explained by variations in the aims, methods of data collection, samples, timing and methods of analysis that were undertaken for each study.

A fundamental point of difference between the studies is whether they examine people’s actual experiences and responses to bushfires, or their attitudes and intended responses to future bushfires. Differences in the way questions are worded and the response options that are provided also account for some of the differences. For example, telephone surveys required rapid responses generally according to pre-defined categories, while face-to-face interviews and mail surveys allow respondents to reflect more and provide a wider range of responses.

Small variations would be expected between studies even where they used identical samples, methods and survey questions. This is due to the uncertainties involved in any sampling. For the quantitative surveys reviewed here, the expected variation is generally within plus or minus 5% (at a 95% confidence level, which is the level typically employed in survey research). Greater variation can be expected in qualitative research due to the more subjective and interpretive nature of analysis. The quantitative reports all had adequate sample sizes ranging from about 500 to 1350. However, as different sampling strategies were used some reports were based on people with greater awareness and interest in bushfire risk, which would be expected to return higher scores for awareness, knowledge and preparation.

Conclusions

The review identified a range of common findings across the reports. The reports suggest high levels of bushfire awareness in high risk areas (around 80%), but lower levels in more suburban locations. Note that recollections of the 2009 fires, subsequent media coverage and heightened concern about the 2009/10 fire season are likely to have raised levels of awareness above what they would otherwise have been. The reports also suggest that around two-thirds of households in bushfire risk areas have fire plans, with around one-quarter having a written plan. Importantly, however, there is considerable variation in the quality of people’s plans and preparedness, and therefore in their effectiveness during fires. There appears to be a moderate level of understanding of the new Code Red FDR, with around 60% of residents understanding that ‘Code Red’ refers to the highest level of fire danger and that emergency services’ advice is to leave early on these days. The reports confirm that many residents intend to wait for advice from emergency services or until they are directly threatened before taking action. Around one-quarter of those who intend to leave would wait for advice from emergency services.

The disparity between intentions and actions is a major issue for fire and emergency services. Research demonstrates that what people intend to do and what they actually do during a bushfire can vary considerably, with actual responses often more risky. It is clear that understanding and good intentions do not necessarily equate to and are not good predictors of appropriate actions. Fire and emergency services must continue to raise awareness and promote better planning and preparedness in bushfire risk areas, especially for the minority who remain unaware of the risk. However, future research should concentrate on resolving the very large gap between good intentions and appropriate actions.

Acknowledgements

We thank the Office of the Emergency Services Commissioner for funding the research reported here. In particular, we would like to acknowledge the support of Brian Hine and others who reviewed and provided feedback on the report. We also thank reviewers from the CFA and Bushfire CRC for their comments. Other researchers from the Centre for Risk and Community Safety at RMIT University made a significant contribution to the report on which this paper is based, including Kate D’Ambrosio, Dr Katharine Haynes, Adriana Keating and Andreana Reale.

About the authors

Dr Joshua Whittaker is a NCCARF Research Fellow in Emergency Management at RMIT University. He completed his PhD on community vulnerability to bushfires in 2008. Since then he has worked on a range of projects for the fire and emergency management sector. Joshua took a lead role in the Bushfire CRC’s research into human behaviour and community safety during the 2009 ‘Black Saturday’ fires.

Professor John Handmer leads RMIT’s Centre for Risk and Community Safety and the University’s Human Security Program. He was Research Leader for the Bushfire CRC’s Program on self-reliant communities and is currently Principle Scientific Advisor for the CRC. He is also Convener of the National Climate Change Adaptation Research Network for Emergency Management. With Damien Killalea of the Tasmania Fire Service he led the Black Saturday human behaviour research program.

References

Reports reviewed (authorship is acknowledged where known):

Office of the Emergency Services Commissioner: Where are they going? People movement during bushfires AND Household locational intentions during bushfire threats. Melbourne: OESC / Strahan Research. (Author: Robyn Betts)

Department of Justice: Summer fire campaign: benchmark research AND Wave 1 research. Melbourne: Department of Justice.

Country Fire Authority: Behaviour and intentions of households in high bushfire risk areas. Melbourne: CFA / Strahan Research.

Behaviour and intentions of households on code red days. Melbourne: CFA / Strahan Research.

Bushfire CRC: Research results from the February 7th Victorian fires: first report on human behaviour and community safety issues. Melbourne: RMIT University / Bushfire CRC. (Authors: Joshua Whittaker, Glenn Elliott, John Gilbert, John Handmer, Katherine Haynes, Jim McLennan and Sean Cowlishaw)

Research results from the February 7th Victorian fires: second report on human behaviour and community safety issues. Melbourne: RMIT University / Bushfire CRC. (Authors: Joshua Whittaker, Katherine Haynes, Jim McLennan, John Handmer, and Briony Towers)

Use of informal places of shelter and last resort on 7 February 2009. Melbourne: La Trobe University / Bushfire CRC. (Author: Jim McLennan)

Other references:

Department of Human Services. 2009. ‘January 2009 heatwave in Victoria: an assessment of health impacts’. State Government of Victoria. http://www.health.vic.gov.au/chiefhealthofficer/downloads/heat_impact_rpt.pdf [Accessed 28 July 2010]

Eysenck H.J. 1995. ‘Problems with meta-analysis’. In: Chalmers I., Altman D.G. eds. 1995. Systematic reviews. London: BMJ Publishing Group. pp. 64-74.

Greenhalgh, T. 1997. ‘How to read a paper: papers that summarise other papers (systematic reviews and meta-analyses)’. BMJ 315, pp. 672-75.

Handmer, J., O’Neil, S. and Killalea, D. 2010. ‘Review of fatalities in the February 7, 2009, bushfires’. Bushfire CRC. http://www.bushfirecrc.com/research/downloads/Review-of-fatalities-in-the-February-7.pdf [Accessed 28 July 2010]

Haynes, K., Handmer, J., McAneney, J., Tibbits, A., and Coates, L. 2010. ‘Australian bushfire fatalities 1900-2008: exploring trends in relation to the ‘Prepare, stay and defend or leave early’ policy’. Environmental Science and Policy 13 (3), pp. 185-94.

Hemmingway, P. and Brereton, N. 2009. ‘What is a systematic review?’ http://www.whatisseries.co.uk/whatis/pdfs/What_is_syst_rev.pdf [Accessed 28 July 2010]

Proudley, M. 2008. ‘Fire, families and decisions’. Australian Journal of Emergency Management 23 (1), pp. 37-43.

Teague, B., McLeod, R., and Pascoe, S. 2009. 2009 Victorian Bushfires Royal Commission: interim report. State Government of Victoria: Melbourne.

Tibbits, A. and Whittaker, J. 2007. ‘Stay and defend or leave early: policy problems and experiences during the 2003 Victorian bushfires’. Environmental Hazards 7 (4), pp. 283-90.

Whittaker J. and Handmer J. 2010. Review of key bushfire research findings. OESC and RMIT University. http://www.royalcommission.vic.gov.au/Documents/Document-files/Exhibits/WIT-3007-001-0041 [Accessed 28 July 2010]

Footnotes

1 Following the February 2009 fires, AFAC convened a national workshop to review the fire danger rating (FDR) system and accompanying warning messages. The new highest fire danger rating is called ‘Catastrophic’ or ‘Code Red’.

2 The Department of Justice and OESC both produced two reports from their respective research projects. Our review considers each pair concurrently.