Can major post-event inquiries and reviews contribute to lessons management?

Abstract

Significant disaster and emergency management events are invariably followed by formal post-event inquiries and reviews. Such reviews identify lessons to improve future capacities and set the agenda for policy and management reform for emergency management organisations. As a result, there is a substantial body of reflections and recommendations gathered across all hazard types and jurisdictions by formal, structured inquiry processes that contribute to lessons management for the emergency sector. However, whether there is any coherence or core lessons emerging for the Australian sector from the totality of post-event inquiries is unknown. The work reported here identifies the recommendations from these inquiries. A meta-analysis of 1336 recommendations made in 55 Australian major post-event reviews and inquiries since 2009 revealed common themes. The recommendations were compiled into a comprehensive database and categorised into 32 themes. The analysis highlighted recurrent themes from recommendations spanning multiple jurisdictions. The study indicates the potential value for Australian and New Zealand emergency management agencies and jurisdictions of using the aggregate data organised as a resource for lessons management.

Introduction

Lessons management is an important and increasing focus of organisational theory and practice in a generic sense (e.g. Milton 2010) and in specific sectors such as industrial accidents (e.g. Kletz 2001), the health consequences of accidents and disasters (e.g. Savoia et al. 2012) and in emergency management (e.g. Donahue & Tuohy 2006, Handmer & Dovers 2013). In Australia, the Australian Disaster Resilience Lessons Management Handbook (AIDR 2013) summarises the importance of the task, sets out general frameworks and principles for lessons management and provides further resources and references. As the singular and authoritative source  for lessons management in Australian, the handbook serves to place the question of post-event inquiries and lessons management in context. The handbook states (emphasis added):

‘Lessons management’ is an overarching term that refers to collecting, analysing and disseminating experiences from operations, exercises, programs and reviews … Interoperability of lessons management systems across agencies, sectors and jurisdictions will facilitate information sharing and national analysis.
(AIDR 2013, p. 1)

An investigation of the large corpus of material of possible relevance to lessons management included collecting information, reviews and cross- sectoral and jurisdictional experiences at the national scale. AIDR (2013, p.15) identifies post-event reviews as a ‘collection opportunity’, however, the potential of the totality of this resource remains unexplored to date. The handbook describes four steps: collection, analysis, implementation and monitoring and This paper deals largely with the first step to establish if post-event inquiries, as communicated through their all-important recommendations, represent a coherent source of issue and reform identification and result in lessons management at an aggregate scale.

Significant natural disasters and emergencies in Australia are almost always followed by formal, complex, post-event inquiries and reviews (inquiries). These inquiries vary in form and focus, however, the common objective is to identify the cause and consequences of disasters and recommend future practices for better outcomes. In some cases, they attribute responsibility or blame for failings.

Formal reviews and inquiries are an important aspect  of lessons management (Eburn & Dovers 2015, 2017b). They provide opportunities for identification and learning of lessons relating to how the emergency management sector, including governments, business and individuals can better prepare for, respond to and recover from emergency events. In particular, there is significant interest in understanding how the findings, and the recommendations that distil those findings into suggested actions from formal reviews, can drive continuous improvement by emergency services agencies and others.

The outcomes of major inquiries in one jurisdiction sometimes have ramifications and lead to reform action in other states and territories. For example, following  the 2009 Victorian Bushfires Royal Commission, there was widespread consideration of findings and recommendations by interstate emergency management agencies. This consideration led to national initiatives  such as revisions to the fire danger rating system and amendments to the Australasian Inter-service Incident Management System. While this may occur in the wake  of larger, high-profile events and inquiries, it is unknown whether there are recurring themes and important lessons for the wider emergency management sector in other post-event inquiries. This is independent of whether a jurisdiction- and hazard event-specific inquiry makes recommendations that lead to reform and thus contribute to lessons management. Formal inquiries require a vast amount of effort, both by those who undertake them and those who respond to them. Given their frequency, it is prudent to consider the totality of recommendations in aggregate to permit a comprehensive view of consistent issues.

Study purpose

The purpose of this study was to generate a high- level description of the major recurrent categories of recommendations across multiple post-event reviews conducted in Australia since 2009. The following negative hypothesis was provided to the review team:

There are no common themes to be identified when comparing and contrasting major post-incident reviews of emergency incidents, and the outcomes of those incidents and consequent recommendations turn on their own particular facts.

Testing this negative hypothesis seeks to understand whether there is ongoing value for Australian emergency services organisations in considering the lessons from major reviews and inquiries from other jurisdictions, or whether lessons are too specific and lack broader import. Importantly, this study looked at whether the large  corpus of inquiry recommendations is worth considering and organising and using as a national resource of lessons management material.

Method

A core element of this study involved preparing a comprehensive and user-friendly database of recommendations from post-event reviews and inquiries. This can be used to inform lessons identification  practices at organisational and,  potentially,  national policy and management levels.

This review updated and developed earlier work by Eburn and colleagues (2014) that considered the recommendations from bushfire-related inquiries occurring over 75 years (1939-2013). A desktop search revealed that more than 140 reviews and inquiries were undertaken since 2009. The list was restricted to a subset of 55 inquiries by applying criteria to exclude narrow technical or legal inquiries, or those with no recommendations or recommendations that lacked wider relevance. In total, 1336 recommendations were added into the database. Appendix A in Cole and co-authors (2017) provides the full list of inquiries, their type (coronial, agency, independent, etc.), jurisdiction, hazard focus and number of recommendations (summarised in Figure 1). The database is usable in being searchable  on the basis of themes, inquiry type, year, jurisdiction  and hazard type. The database links to digital copies of inquiry reports allowing further investigation into the context within which recommendations were generated.

Figure 1: Australian inquiries by type and state and territory jurisdiction since 2007. Source: Cole et al. 2017.

Figure 1 is a graphical breakdown of the types of inquiry included in this review and the jurisdiction in which each was undertaken.

Each recommendation was independently coded  into one of 32 themes by three members of the review team. Where there was not complete agreement regarding the coding of a recommendation, each reviewer’s interpretation was discussed and the final code was agreed by consensus. Coding was initially  based  on the categories generated in Eburn and colleagues (2014), with additional categories developed as coding progressed. A small number of recommendations were difficult to allocate to themes; however, these were relatively few. The commonality of major themes across inquiries suggests a robust categorisation, particularly  as no theme was covered by only one inquiry and most were covered by more than ten inquiries.

Table 1 shows the themes and the distribution of the themes and recommendations across the 55 inquiries.

A targeted approach to thematic analysis was employed to manage time and resource limitations while providing robust investigation of the negative hypothesis. Initial analysis of recommendations was restricted to:

• the five most common themes
• several themes containing an average number of recommendations
• the five themes containing the least recommendations.

This developed an initial understanding of the main messages (or lack thereof) within a range of themes to confirm that the themes represented reasonably coherent sets of issues, also reported in Cole and co-authors (2017).

Table 1: Major descriptive themes and number of recommendations for inquiries reviewed.

Note: Reviews and inquiries may have recommendations that relate to multiple themes. 

Results and discussion

The analysis demonstrated that  the  proposed negative hypothesis is false. A significant number of recommendations were identified that are matched by similar recommendations in different jurisdictions. The analysis also revealed a number of recommendations that were not matched by similar recommendations, but were generic in nature and could have broader significance for other jurisdictions.

This study proves the viability of the approach developed here for agencies and the sector to identify and understand the themes and recommendations from major post-event reviews that may be relevant   to their jurisdictions. It also provides the means to do this through the preparation of the usable database. In Milton’s (2010) terms, ‘after-action reviews’ (p. 54) are a viable ‘formal collect system’ (pp. 28-29) of information for lessons management, and that, fashioned into a database, post-event inquiries are ‘lesson repositories’ (p. 103).

Cole and colleagues (2017) provide discussion of the observations relating to major themes, as well as areas that received less focus than might be expected given their standing in public policy and research. Some broad observations show the potential for further investigation of particular themes, the relevant recommendations and the context in which they were made.

Distribution of recommendations

A number of themes, while raised across multiple  inquiries, were dominated by one inquiry. A prominent example is the ‘Land use planning, development and building codes’ theme that includes 81 recommendations. Of these, 52 recommendations were made by the Queensland Flood Commission of Inquiry and a further 11 recommendations derived from the 2009 Victorian Bushfires Royal Commission. Another example is the ‘Access to fire grounds’ theme that arose largely out of the Post Incident Analysis Bridgetown Complex (Government of Western Australia 2009). These cases highlight that the magnitude of some themes may appear exaggerated where a singular review has resulted in many recommendations relating to a specific theme. Therefore, caution should be exercised when drawing broad conclusions from aggregated data. What is a prominent issue in one jurisdiction (producing many recommendations) may not be as important in other jurisdictions for a range of contextual reasons including geographies, climate, timing, institutions and emergency management arrangements. However, it is possible that an event and inquiry exposes issues and potential policy and management actions relevant to other jurisdictions, but which have not yet been revealed via a similar event or inquiry.

An important factor to note is that the scope and limitations of inquiries are generally determined by the TOR. This may impact on the recommendations put forward by any one inquiry. This was not examined in detail by this study.

Prominent themes and notable absences

The following themes, or groups of themes, were the major focus of recommendations.

‘Doctrine, plans and standards’ theme was the largest with 200 recommendations, focusing primarily on the processes and practices within the emergency management sector. In combination with the themes of ‘Incident management teams’, ‘Emergency management agency organisation, management and authority’ and ‘Training, skills and behaviours’, the focus on organisation and function of emergency management agencies was apparent and, arguably, to be expected.

Better coordination between emergency management agencies is often recommended. When combined with the ‘Whole-of-government’ theme, these recommendations suggest a need to dismantle silos between emergency management agencies to improve policy and management integration across the sector.

‘Community warnings and communication’ theme was the focus of 74 recommendations and ‘Community education’ theme the focus of 57 recommendations. The combination of these themes highlight the important role that government is expected to play in preparing and delivering educative materials, information and warnings to communities effected by emergencies.

As a cautionary note, some themes that warn against treating the aggregated recommendations and focus of post-event inquiries as a singular agenda of important issues and actions for emergency management, reflect the limits of post-event inquiries to cover all-important matters.

First, some themes that are reasonably prominent might nevertheless be expected to feature more strongly. ‘Relief and recovery’ theme for example, could be expected to feature more than it does. A possible explanation for the apparent lack of attention is the timing of inquiries, which may occur before issues with recovery and relief emerge in full. Another is ‘Hazard reduction burns’, which with 36 recommendations from 12 inquiries, is less than might be expected given the highly contested nature of that practice and constant attention the topic receives.

Second, given the importance of some topics within emergency management and disaster policy, the lack of attention from inquiries towards a number of themes is notable. The following themes received less consideration:

• ‘Volunteers’ theme had 23 recommendations from nine inquiries. Recommendations relating to this theme are surprisingly scarce given that emergency management capacity in Australia is heavily reliant on local volunteer fire brigades and state emergency service organisations.
• ‘Occupational Health and Safety’ was a minor theme, with nine recommendations from five inquiries. The lack of attention is noteworthy given disasters are by definition dangerous for staff and volunteers, incidents are not uncommon and, legally, there is a greater responsibility laid on agencies and senior executives for their staff than there is for the public generally.
• ‘Funding’ and resourcing constraints will always limit the capacity of agencies to do their work, including implementing inquiry recommendations. Nevertheless, funding is only a minor theme within the It is possible that those undertaking inquiries perceive comments on relative budget allocations by governments as outside the purview of their role.

Perhaps the most prominent gap is the minimal attention given to the roles and responsibilities of non-government actors.

The role of government within shared responsibility

The greatest focus of recommendations was on the role of the government within the emergency management sector. Greater focus on the role of government appears to overshadow the focus of the inquiry on other important themes. The most striking example is the lack of attention relating to the key actors within the policy goal of shared responsibility.

Shared responsibility stresses the complementary roles of government, communities, individuals,  households and the private sector. The concept was prominent in  the Victorian Bushfires Royal Commission process, and has since been articulated in Australian policy through the National Strategy for Disaster Resilience (Attorney- General’s Department 2011) and elsewhere, as well as featuring in the Sendai Framework for Disaster Risk Reduction 2015-2030 (UNISDR 2015).

Despite the importance of shared responsibility as a policy direction, inquiries offer little attention  to  the topic, or otherwise comment only in a general fashion without issuing tangible recommendations. For example, only a small number of recommendations target  members of the community (including individuals and households) through the theme ‘Personal responsibility’. Recommendations also gave scant attention to the use  of local knowledge and the role of business and industry. Together, the lack of attention that these topics received indicate a disconnect between the focus of inquiries and emerging policy discourse.

Formal reviews and inquiries are typically guided by Terms of Reference (TOR) that define the scope and limitations that must be adhered to during the review process. A simple reason for the disjunct  could  be that the TOR restricts reviewers to investigate certain matters. While the TORs of reviews were not examined  in detail during this study, a cursory look suggested that they are general enough to allow inquiries to head in any direction.

Natural disasters and emergencies are high profile public events and decisions made in response are likely to have (sometimes adverse) political implications (Eburn & Dovers 2017a). Inquiries are also commissioned and often undertaken by government and its agencies. For this reason, it may be politically sensitive to lay expectations, let alone blame, on the community. This  may be especially true if it is likely that the outcomes of inquiries will have a negative effect on the government or agencies.

In addition, the political nature of inquiries also applies  to the procedural aspect of evidence gathering that feeds into recommendations. Inquiries  will  consider and respond to issues and information put before them through submissions and, in some cases, before a judiciary. Under these circumstances it is unlikely that emergency management agencies or the public will target affected communities on the issue of shared responsibility in the wake of events that have revealed their vulnerability.

It may also be easier, and thus more effective, to target recommendations at specific agencies  and their functions rather than the more amorphous and diverse ‘community’ or ‘private sector’. If the purpose of inquiries is to identify the cause and consequences of emergencies, and to set the agenda for reform to policy and practice in the sector, then it follows that they would do this via the most effective means. The role of government in emergency management is generally well defined and widely accepted by the public. Governments have a clear mandate, and in most cases greater funding, to respond to recommendations. Therefore, there may be a perception that recommendations targeted at the government are more likely to be adopted and implemented than those directed at other actors.

Recommendations database

An objective of this study was the creation of a comprehensive database that is a valuable resource for gaining an overview of, and insight into, the recommendations that are made across multiple jurisdictions, hazards and inquiry types. Given the value of the material described and reviewed, the database is an important tool to support increased inter-jurisdictional learning and lesson sharing.

The Bushfire and Natural Hazards CRC (CRC) owns the database and will maintain the data to ensure it remains current and accessible. The database can assist jurisdictions and policymakers to identify and consider recurring recommendations and themes within their operating and risk environments. The database will be hosted by the CRC through a publicly accessible web-based platform from mid-2018.

Conclusion

Against the background of increasing attention  to lessons management in Australian emergency management, this study revealed the presence  of consistent themes across multiple post-event inquiries since 2009 and the value of the emergency management sector in considering the totality of inquiry recommendations. A usable database of inquiry recommendations has been developed. While recommendations are made within the context of  specific jurisdictions, there are multiple recurrent recommendations revealed in the dataset. This suggests that there are opportunities for emergency management agencies to learn and benefit from inquiries and from the cumulative insights into a particular theme across inquiries over time. Uptake and continued use of the database by emergency management agencies and others can support lesson management practices to:

• identify and understand the themes and recommendations from major post-event reviews that may be relevant to their jurisdictions or to the sector as a whole
• track a jurisdiction’s progress towards implementation of recommendations
• identify themes from other jurisdictions and review their systems to consider whether similar recommendations would be likely to occur.

There is strong potential for deeper investigation into particular issues revealed as recurring or prominent in post-event inquiries. Recommendations are both calls to action and a form of ‘index’ that provides detailed description and discussion in inquiry reports that lead to recommendations. Research and analysis based on multiple post-inquiry reports would be worthwhile into matters such as training in the sector, inter-agency collaboration, cross-portfolio policy and response capacities and urban planning. These issues are recurrent in inquiries and central to contemporary debates in emergency management.

While caution is required against applying recommendations in a wholesale manner to another jurisdictions, this approach provides a broad indication of the topics that may be worth considering in more detail and in a jurisdictional context. Although the information described here and organised in the forthcoming database is not a complete picture of issues, lessons and actions, it represents a significant and previously untapped input to lessons management and a substantial improvement on the sector’s previous capability to manage lessons from across multiple jurisdictions, hazard types and years.