the Creative Commons Attribution 4.0 License.
the Creative Commons Attribution 4.0 License.
| 13 Mar 2019
ANYCaRE: a role-playing game to investigate crisis decision-making and communication challenges in weather-related hazards
Galateia Terti
Isabelle Ruin
Milan Kalas
Ilona Láng
Arnau Cangròs i Alonso
Tommaso Sabbatini
Valerio Lorini
This study proposes a role-playing experiment to explore the value of modern impact-based weather forecasts on the decision-making process to (i) issue warnings and manage the official emergency response under uncertainty and (ii) communicate and trigger protective action at different levels of the warning system across Europe. Here, flood or strong-wind game simulations seek to represent the players' realistic uncertainties and dilemmas embedded in the real-time forecasting-warning processes. The game was first tested in two scientific workshops in Finland and France, where European researchers, developers, forecasters and civil protection representatives played the simulations. Two other game sessions were organized afterwards (i) with undergraduate university students in France and (ii) with Finnish stakeholders involved in the management of hazardous weather emergencies. First results indicate that multi-model developments and crowdsourcing tools increase the level of confidence in the decision-making under pressure. We found that the role-playing approach facilitates interdisciplinary cooperation and argumentation on emergency response in a fun and interactive manner. The ANYCaRE experiment was proposed, therefore, as a valuable learning tool to enhance participants' understanding of the complexities and challenges met by various actors in weather-related emergency management.
Extreme weather and climate events challenge weather forecasting and emergency response operations and are often related to high social, environmental and economic impacts worldwide (WMO, 2015). In Europe, recent examples of devastating weather events include the 2003 European heatwave from which France suffered the worst losses with almost 15 000 deaths from 1 to 20 August (Hémon and Jougla, 2004), the 15–16 June 2010 flash-flood event in the Var Department in France that caused the loss of 26 people (Poussin et al., 2015; Ruin et al., 2014), and the catastrophic fires in the forests of central Portugal that killed 64 people and destroyed more than 480 houses on 17 June 2017 (Mayer et al., 2017).
It is apparent that modern technological improvements including increases in accuracy and lead time of the hydro-meteorological forecasts alone do not guarantee reduction of fatalities and economic disruption (Petrucci et al., 2018; Terti et al., 2017). Effective disaster risk management relies not only on the accuracy and precision of official hazard predictions and related warnings issued by forecasters but also on how those are communicated to and interpreted by end users to support informed decision-making on allocating human and material resources before and during the crisis (Kox et al., 2018). When a severe-weather phenomenon such as a big storm threatens an area with flooding within a European country, forecasters in governmental forecasting offices use raw meteorological data and model outputs to run hydrological models and inform emergency services and other user groups such as road maintenance services for the imminent risk to life and property in the area of their responsibility. This information may be in the format of text messages or visual warnings and are transmitted to emergency managers in the control centres and are further disseminated to other public authorities (e.g. local fire stations) or voluntary organizations to prepare for action. Before deciding on any emergency action, emergency managers examine the forecast information and deal with uncertainties seeking the most concrete indications for both the weather event and its potential impacts. In the words of the World Meteorological Organization (WMO), the emergency management decisions need to be supported by the knowledge, not only of “what a hazard will be” but also of “what a hazard will do” (WMO, 2015). In addition, the Sendai Framework for Disaster Risk Reduction (2015–2030) calls for an approach that is multi-hazard (i.e. takes into account the interaction of natural and man-made hazards), people-centred (i.e. takes into account the needs and rights of the affected persons) and preventive (i.e. aims to completely avoid the potential adverse impacts of a disaster through action taken in advance) (Müller et al., 2017).
In this direction, recent decision-support tools promote the elaboration of multi-hazard impact-based or risk-based forecasts that translate meteorological and hydrological hazards and related cascading effects into sector- and location-specific impact estimations to improve responses and the public understanding and coping capacity to those risks (Luther et al., 2017). It is hypothesized that impact-based information systems integrating social vulnerability and behavioural processes in the forecast-warning system will help emergency services to better capture and respond to life-threatening situations and catastrophic scenes emerging from the conjunction of the hazard and social vulnerabilities that evolve in space and time (Creutin et al., 2013; Ruin et al., 2008; Terti et al., 2015, 2019).
To take a first step towards exploring this hypothesis, we propose a new role-playing experiment that engages participants in the decision-making process at different levels of the weather-related emergency system (from hazard detection to citizen response). The focus of the experiment is a tabletop or pen-and-paper role-playing game (PnP) for adults in which participants act out their role through speech while sitting in a comfortable setting (Cover, 2005). The PnP game is structured in progressive simulations in which improved multi-model outputs, including information on (i) impact assessments and maps and (ii) live data on exposure and vulnerability derived from social media and crowdsourcing (called “impact-based vulnerability information” hereafter), are presented as new decision-support tools for the players representing an emergency group. The simulations are built based on the hypotheses that dynamic real-time impact information (e.g. potentially affected population and critical infrastructure, economic damage) can support emergency services to
- i.
spatially and temporally locate critical spots for intervention and, therefore, better allocate available resources to protect lives and livelihoods;
- ii.
communicate more targeted warnings and emergency guidance messages to help the public understanding of how certain hazards may affect their lives, livelihoods and properties, leading to appropriate self-preparedness and self-protective actions.
An important advantage of the simulation approach is its dynamic nature that allows participants to experiment with real-time decisions and experience potential changes in the outcome over time (Pasin and Giroux, 2011). By first providing the players with hazard-forecast information alone and then adding impact-based forecasts at each round of the game, it allows them to progressively integrate the use of impact-based model outputs and reflect on the usefulness of such information in supporting the collective emergency decision-making process. This “learning by doing” process – a fundamental principle in experiential learning theory (Kolb, 1984) – has the privilege of taking place in an informal setting without real consequences.
Role-playing games (RPGs) are virtual simulations of real-world events especially designed to educate, inform and train players for the purpose of solving a specific problem (Bowman, 2010; Drachen et al., 2009; Susi et al., 2007). In this study, ANYWHERE (http://anywhere-h2020.eu, last access: 11 March 2019) Crisis and Risk Experiment (ANYCaRE) is developed with the aim of providing a wide range of audiences with the opportunity to enhance their understanding of the recent uncertainties and dilemmas embedded in the real-time warning and emergency response processes. Through realistic what-if scenarios, the players of the ANYCaRE game are invited to assess different information describing an imminent risk situation and to decide collectively what protective actions, if any, are needed. Given the scarcity of chances to actually experience demands for decision-making on weather emergencies, playing the simulation-based game is a simple but essential way for participants to nourish their recognition of the emergency management difficulties and beneficial tactics (Crichton et al., 2000). On the other hand, playing may provide an excellent opportunity for current or future decision makers to train on dynamic and uncertain incidents that may occur in times of weather-related crises.
The modern concept of a serious game was first introduced by Abt (1970), who viewed computer- or pen-and-paper-based games as educational tools to be used both inside or outside of classrooms (Djaouti et al., 2011). Originally such games have been used for training in the military, but have since expanded to many domains (Mayer, 2009; Djaouti et al., 2011). Although evaluating the effectiveness of role playing or serious computer-based games is not always a straightforward task (Backlund and Hendrix, 2013), the literature values such educational or training games as motivating experiential learning tools that go beyond traditional passive learning approaches often applied in conferences and seminars (Boyle et al., 2014; Dieleman and Huisingh, 2006; Salas et al., 2009; Solinska-Nowak et al., 2018).
In the field of disaster risk management, role playing has been successfully used to increase public awareness as well as to promote preparedness and prevention of losses (Rebolledo-Mendez et al., 2009). Examples of applications include role-playing simulations on flood management on cultural heritage sites for university students (Huyakorn et al., 2012), serious online games on natural disasters (e.g. tsunami, earthquakes) for children (Pereira et al., 2014), and serious board games on resilience to geological hazardous events for school students or adult stakeholders (Mossoux et al., 2016).
In the domain of emergency response and crisis management, exercises, drills or simulations serve local authorities, governmental agencies or private organizations in testing systems and procedures and training the staff in the management and coordination of responses to disruptive and potentially disastrous events (Lee et al., 2009). Effective designs of scenario-based crisis management game simulations have been proposed to create opportunities for emergency trainees to rehearse crisis decision-making and prepare for real emergencies (Friman, 1991; Metello et al., 2008; Walker et al., 2011; Oulhaci et al., 2013; Alharthi et al., 2018). Because crisis work involves response teams from different organizations, simulations tend to be thought to test not only the responsiveness of a specific body, but also the capability of different forces to coordinate their activities (Di Loreto et al., 2012). Most crisis situations involve dealing with wicked problems, where solutions are not true or false, but better or worse. Di Loreto et al. (2012) see crisis management as a time-constrained activity comprised of both predictable and unpredictable elements that require a person to dissect problems, make plans for each sub-problem, integrate local actions with global ones (avoiding solving one problem locally that create one at a larger scale) and have knowledge of the territory's vulnerability, communication and cooperation skills together with a clear definition of roles and responsibilities. The overall objective is to “quickly converge toward a normalization of the situation” (Di Loreto et al., 2012, p. 2) using predictive models and plans to anticipate foreseeable events and soft skills allowing the necessary flexibility and capabilities to overcome unexpected events (Robert and Lajtha, 2002). With respect to crisis and disaster risk management, Van Ruijven (2011), Di Loreto et al. (2012), Alharthi et al. (2018) and Solinska-Nowak et al. (2018) proposed reviews of serious games and simulations, specifically looking at the challenges of using serious games for research and assessing their effectiveness in soft-skills training. If the ANYCaRE game falls in the most common category of face-to-face multi-player experience with lively interactions between players, it is one of the few role-playing game dealing with early warning systems (EWSs) and allowing to test new forecasting products (Solinska-Nowak et al., 2018). With a similar purpose the UK Met office developed the Forecast-based early action game, simulating a community affected by floods who need to prioritize their actions based on a severe-weather forecast (UK Met Office, Deltares, and Red Cross Red Crescent Climate Centre, 2018). Another role-playing game called Adapted Technologies for Early Warning Systems: Playing with Uncertainty aims to illustrate the complexity of decision-making in EWSs and the need to integrate all its components by playing with different scenarios of economic development, stakeholder participation, technology and uncertainty (Garcia Londoño and Fearnley, 2018).
According to Bowman (2010), the role-playing method (i) “enhances a group's sense of communal cohesiveness by providing narrative enactment within a ritual framework”, (ii) “encourages complex problem-solving and provides participants with the opportunity to learn an extensive array of skills through the enactment of scenarios”, and (iii) “offers participants a safe place to enact alternate personas through a process known as identity alteration”. Therefore, it is hypothesized that the methodological approach adopted in ANYCaRE allows the following:
-
Argumentation on weather crisis management should be collaborative (Huang et al., 2010). Dialectic reflection on weather uncertainties and challenges helps participants to form their situational awareness and build a common strategy to solve problems of safety during extreme weather events. Therefore, the experiment facilitates collaboration and coordination between participants who may have distinct fields of expertise and/or belong to different national or local institutions across Europe.
-
Training of decision-making skills for emergency management is needed (Linehan et al., 2009b). Through progressive simulations the players are expected to get more and more familiar with good practices in emergency management. Serious games are recognized in the literature as useful tools for training since they offer an environment in which trainees can experience demands of emergency management under stress before the real crisis (Crichton et al., 2000). The reception of new data as the game progresses makes the player cultivate soft skills such as communication and understanding of auxiliary or inconsistent information in a limited time frame (Linehan et al., 2009a).
-
Relevant behaviours for emergency response are fostered (Meesters and Van De Walle, 2013). A safe playing environment in which participants act out given roles (sometimes very different from their duties in daily life) permits them to gain a deeper understanding of the weather-related risks and decision-making complexities (Rebolledo-Mendez et al., 2009). During the experiment the player can realize conflicting requests arising in times of weather crisis and reconsider the relevance of specific (self-) protective actions.
Cognitive research indicates that, in order for serious games to serve as effective educational means, they should embed the learning outcomes within the game mechanics and provide direct and specific feedback to the actions taken by the players (Bogost, 2007). The learning outcome of ANYCaRE is the improved weather risk-related decision-making for emergency response through modern multi-source inputs. Thus, participants get informed about new forecast products (like multi-hazard or impact-based model outputs) and informational systems (also involving crowdsourced information), and then practice their understanding and interpretation of those tools through playing a realistic crisis scenario. Therefore, the game presents an environment in which it is beneficial to communicate with others to exchange complementary or contradictory information presented to each role and consider competing demands before deciding. The learning outcome is embedded in the gameplay by providing specific feedback on how well the players are performing immediately after decision-making in the various states of the game. During the game, at least one of the game developers acts as an observer of the discussions and guides the decision-making. In addition, at the end of the game, a specific time is dedicated to debriefing players about the experience. With the use of such qualitative methods the experiment aims to assess if and how improved multi-model (and potentially multi-hazard) outputs, including information on impact-based vulnerability data, can support the decision chain in European warning systems towards better responses.
The targeted audience for ANYCaRE varies from students and young researchers to developers, potential users and other stakeholders from different European agencies. The paper presents the first implementations of the experiment in the emergency management of flood (and flash flood) and strong-wind events. First, we describe the research methodology and the conceptual framework for weather-related crisis management adopted to design ANYCaRE. Section 2 explains also how ANYCaRE is set up and portrays general guidance for the design of role-playing experiments including simulations for weather-related crises. Then, we provide details on the storylines and the roles selected for the flood and strong-wind simulations, respectively. Section 4 presents the first applications of the game tested in different contexts. The first results from these experiments are reviewed based on the players' opinions of the gaming experience but also on the modern products provided as input data to support decision-making in the simulations, providing insights into related limitations. Finally, further advancements, extensions of ANYCaRE and opportunities for future applications are discussed.
2.1 General concept and hypotheses
2.1.1 Principles of tabletop role-playing games (TRPGs)
TRPGs are the original form of role-playing games and in principle they are conducted through discussion within an interactive and collaborative storytelling system (Aylett et al., 2008). The players act out the roles of characters and through the game they decide the actions of their character given certain rules and responsibilities within an hypothetical setting (Cover, 2010). In the ANYCaRE game, participants are invited to play specific characters of the decision-making chain (i.e. forecasters, emergency managers or public) in an interactive story related to a weather hazard in a European context. As a narrative-based game, ANYCaRE consists of a storyline and the simulations to be played through various roles. Table 1 describes the key components (storyline, roles, simulations) necessary for the definition of the scenarios and summarizes the questions that guided our design choices and further considerations for each of those components. This table may serve as a design guide for future developers of educational or training exercises in the domain of natural hazard management. For instance, it could help developers to consider options with respect to choices in the real world vs. virtual territory or the time frame and pace of the decision-making related to the type of hazard considered.
Table 1Description of the game components and the corresponding design choices to be made when developing ANYCaRE.
The term “game” mainly refers to an array of simulations that represent certain real-event situations, which are played and manipulated by the participants. The players are guided through the simulations to experiment with the outcomes of their decisions and learn from their playing experience (Dieleman and Huisingh, 2006; Huyakorn et al., 2012). A central function in TRPGs is the games master or moderator (GM), who leads the storytelling during the game simulations (Heinsoo et al., 2008). The GM acts as organizer, arbitrator and moderator of the multiplayer role-playing game and, therefore, they should know the game in detail and be able to answer questions regarding its rules. The games master's responsibilities could be split among two or three people (e.g. designers or organizers of the experiment). The basic role of the GMs is the same in almost all traditional role-playing games, although differing rule sets make the specific duties of the games master unique to that system (Tychsen et al., 2007). For example, in the ANYCaRE game the GM is responsible, among others, for providing feedback to the players about the hydro-meteorological observations, highlighting the relevant safety decisions that should have been taken at each playing round (similar to a weather reporter).
2.1.2 Conceptual framework for weather emergency management
The roles to be played and the potential decisions and actions to be chosen by the players in ANYCaRE are predefined based on qualitative evidence gathered during European workshops that took place in March and April 2017 (Müller et al., 2017) and in previous research (Ruin, 2007). In particular, the game was designed in order to be adapted or easily adaptable to most European countries' warning and emergency decision-making contexts. Based on examples of warning systems from Switzerland (Canton of Bern), Spain (Catalonia), Italy (Liguria), Finland (South Savo) and France, commonalities of these systems were identified and used to simulate realistically the dynamics of the warning and response processes starting with the detection of a potential weather-related threat and ending with decisions related to the coordination of the emergency response (Fig. 1). This process involves the identification of three major components: (i) the type of actors involved in the warning system and their role in the decision-making process, (ii) the timeline/temporality of the warning phases, (iii) the types of actions and decisions the actors need to take in each phase to ensure people's safety.
Figure 1Conceptual framework of the weather-related European Warning Systems adopted for the design of ANYCaRE role-playing experiment. National, regional or local actors are framed within light, medium or dark grey rectangles. According to the warning/response phase in which they mainly operate, actors and their warning or emergency actions and decisions (snipped-corner rectangles) are presented inside the light (detection), medium (hazard warning) or dark-green boxes (emergency response). The dashed arrows illustrate the flow of information among national, regional and local actors. Adapted after Müller et al. (2017).
Depending on the country and its administrative and legal organization, different types of actors may be involved in the warning system. The responsibility for emergency management is also dependent on the spatial scale of the event. Most often, the municipality is the first level of responsibility and the most important actor of the emergency management as soon as the event does not exceed the capacity or the boundary of this local level. Mayors are in charge of the activation of municipal emergency plans. Nevertheless, regional and national levels support their decisions with weather-related hazard forecasting and warning capabilities (meteorological, environmental or hazard-related expert institutes) and regional emergency management centres and rescue services. In times of emergency, actors with complementary competencies are gathered (at the administrative level of concern: local, regional or national in the case of a state emergency), either physically or remotely, to take decisions on how to best manage the crisis to ensure people's safety. Generally, those emergency operation centres (EOCs) include representatives of civil services as weather/hazard experts, police, firefighters and rescue forces, representatives of municipalities, and infrastructure experts from public or private companies (road, telecommunication, energy suppliers). A representative of the highest authority concerned will act as the leader of the group to organize the discussion and finalize the emergency decisions. The centre functions as the kernel of information, by receiving, checking and sharing information with operational teams as well as deciding upon complex emergency actions that need an holistic view of the situation, coordinating efforts and communicating with the public.
To reflect this standard type of crisis management organization, the ANYCaRE game proposes that decisions should be taken in the context of a simulated EOC gathering in the same room or a choice of the actors cited above. A panel of role descriptions that are distributed among the players (randomly or based on their real-life expertise) describes the tasks and responsibilities that each player has to contribute to the collective decision finalized by the group leader. Some roles, such as the ones of the forecasters or the one of the group leader, may be distributed carefully as they require either strong expertise or leadership. It is clear that the way those roles are played may influence a lot the results of the decision-making process and confidence of the overall group in their collective decisions. Nevertheless, random distribution of the roles may also raise participants' awareness about the capabilities and level of expertise necessary to interpret and act upon such complex, dynamic and uncertain crisis situations.
As shown in Fig. 1, the timeline or temporality of the warning process often relates to three phases (Müller et al., 2017). The first one is a detection phase in which model-based forecasts indicate the first signs of a potential high-impact event. At this early stage (3 to 10 days before the event), forecasts are neither accurate in space nor in time and the potential for impacts is difficult to assess. The second phase starts when the event becomes more probable and the location and timing grow less uncertain. This phase, starting about 2 days before the event, relies on model-based forecasts that are sometimes combined with some first empirical observations. At this stage, the first precautionary measures can be envisaged in order to prepare for the alert phase. In the third phase, the alert one, when the magnitude of the event is almost certain as well as its location and timing, it is time for emergency action. Decisions are mostly based on ground-truth observations made by different operational services involved in the emergency response. The temporality of those phases varies according to the type of hazardous events, as some are less predictable than others, like in the case of flash flooding vs. flooding events. Short-fuse events originating from convective storms, for instance, cannot be detected several days in advance, which contracts the warning and response process in a very short period of time. Based on these observations, the ANYCaRE game addresses this temporal aspect of the decision-making process by proposing a timeline adapted to the pace of the relevant hazard.
2.2 Experiment set-up
2.2.1 Steps of the experiment
The experiment progression can be summarized in four progressive steps: (i) introduction to ANYCaRE, (ii) role assignment, (iii) simulations, and (iv) debriefing as proposed in Table 2. The experiment begins with the description of the setting by the GMs. Then, each player is provided with a certain role defining their responsibilities during the game. Players get a few minutes to become familiar with their role and to introduce it to the rest of the group before the main game simulations start. Following the principles of tabletop role playing, the GM facilitates the playing by presenting key aspects of the story and by triggering relevant discussion among the group of players. The designers' team act as observers of the playing process and based on their observations they facilitate the post-experiment debriefing. Based on the authors' experience, they believe a playing group in ANYCaRE should be preferably composed of 10 to 12 players.
Table 2Progressive steps of ANYCaRE experiment. The proposed durations suggest an experiment that lasts up to 2 h.
2.2.2 Objective of the simulation
The simulation aims to put players in the situation of acting out the roles of the different emergency management representatives gathered in the context of an EOC. As a group they are requested (i) to evaluate the weather-related situation and potential threat based on weather forecast information and their own assessment of the level of exposure, potentially supported by impact-based and crowdsourced information, (ii) to select, from a pre-established list, protective actions (if any) and four communication options that can be taken to best inform the targeted public about those decisions (see Appendix A for the details of the options available). The role of the emergency management group is to keep the population safe and ensure smooth execution of everyday activities in the territory while managing a given budget. In the case of weather uncertainty, this is a challenging task including a set of dilemmas. Deciding to alert the population and pushing them to stop their daily activities to take protective measures in areas not hit by a hazard might create unhappiness and loss of confidence in public authorities. On the other hand, people are looking forward to recreational events such as big festivals; potential cancellation requires careful consideration to preserve people's wellbeing without risking their security. Every decision taken by the emergency management group has a consequence, either in terms of human safety or wellbeing and economic value. The objective of the group is to undertake emergency activities from a predefined decision-reporting list, avoiding actions that might prove to be unnecessary at the end of the game. The proposed list of actions depends on the storyline and the purpose of playing and can be easily adapted to different scenarios to be played by different audiences. Thus, different versions of worksheets are created to fit the game implementation (see Appendix A).
2.2.3 Playing rules
Three rounds of decision-making are played successively to simulate the progression of the hazard from its early detection to its landfall. By using multiple rounds, we allow the players to experience evolving hydro-meteorological facets and test different decision-support tools, which give more and more accurate information, as the event occurrence gets closer. The repetition of the decision-making process over several rounds also helps the players to get better at managing their roles and learn from practice. Nevertheless, based on the dynamic and predictability of the simulated event, the pace of succession of the hazard and/or risk information and decisions to be made in the game can vary to represent a few hours to a few days in real life.
Based on weather-related hazard information, the players simulating the EOC need to take decisions related to the three phases of the hazard progression described in Sect. 2.1.2. At the beginning of the simulation, only weather and/or hydrological model-based forecasts are available for the coming hours or days. The level of uncertainty is still high. Round after round, more precise information including impact-based information is provided to reflect the decrease in uncertainty and the potential imminence of the event occurrence.
With this information, specifically distributed to each role with respect to its own responsibilities, the players first need to interpret and share their specific knowledge before envisaging and deciding upon solutions to use against the potential threat they identified. Based on their collective evaluation of the situation, they have a certain time to choose between 3 types of decisions: (i) stay aware and monitor the situation, (ii) take actions in the context of a warning phase and activate the EOC to take precautionary measures, (iii) activate the emergency plan and proceed to specific safety measures. For each of these types of decisions, several more detailed options are proposed on the decision worksheet (see Appendix A) to help the group decide whether they would provide some generalized advice for safety to the public (e.g. “If inside, move to higher floors”, “Be prepared for electricity disruptions”) or if they would proceed to more detailed emergency orders in specific area(s) of the territory (e.g. “Evacuate immediately”). Collective decisions are recorded by filling up one worksheet for the whole group in each game round. Each round is composed of two trials, one in which only existing basic hydro-meteorological forecasts are available and the second in which additional, more sophisticated decision-support products are provided. The group reports its choices in the relevant trial column on the worksheet at the end of each trial. Since certainty in the warning-response processes is a key element of efficient crisis management, the group is also asked to evaluate the level of confidence associated with their decision.
Figure 2Presentation and brief description of the territory considered in the storyline of ANYCaRE for the (a) flood scenario and (b) strong-wind scenario. Each area of the territories includes attributes for special consideration in the emergency decision-making (e.g. camping, schools, dangerous intersections, industries), representing critical points for intervention in European cities.
To reflect the time pressure that real-life EOCs always face, the players are given a limited time in which to provide responses to each trial and the GM is in charge of pressing the group to obtain their decision in time. For example, in the case of three-round simulations, each trial lasts 10 min in the first two rounds, whereas in the last round each trial lasts 8 min. Therefore, the first two rounds last 20 min each and the third round lasts 16 min (or less if the players are fast to take decisions on a given round). This time includes 1–2 min for the GM to present a short summary of the (hydro-) meteorological situation and consequences of the decisions that have been taken in the previous round. The experiment should preferably last less than 2 h (including the debriefing time). A detailed penalty scheme with costs in terms of safety, emergency resources, wellbeing, etc. may be assigned to each activity listed in the worksheet or not. If the objective of the playing is the training and or information/education then specific penalties may be not necessary. Triggering debate and discussion to enhance understanding is a major priority. If penalties are assigned, the designer(s) should ensure that every action has a straight consequence or penalty and that the initial credits given to the players are adequate, even for the worst-case scenario.
3.1 Flood scenario
3.1.1 Storyline and roles
Inspired by European locations we introduce Anywhere City, an imaginary agglomeration including three distinct areas: A–C, located on the slopes and at the foot of highlands drained by two fast-reaction rivers (Fig. 2a). Area A is characterized by relatively steep tree-covered slopes drained by a small basin (e.g. 265 km2) and is known for its fast response to precipitation. A suburb of about 1000 residents and one school are located on the slopes. There are no permanent settlements or critical infrastructure in the flood-prone zone but one camping ground is located in the forest close to the riverbed (within the 10-year return period flood-prone zone). Area B is composed of both highlands and lowlands drained by a river basin of about 3000 km2. The densely populated urban area (e.g. 100 000 citizens) is located in the lower part of the basin. It includes the majority of schools, hospitals and other public services. About 30 % of the residential, commercial areas and public services are located in the 20-year return period flood-prone zone. Finally, area C is typical lowland with a large floodplain located in the lower part of a larger river basin (up to 4000 km2). There are no permanent settlements in C but the area surrounds the main bridge of Anywhere City, calibrated to resist a 50-year flood. The area is characterized by seasonal agricultural activity and a recreation place where the annual festival of Anywhere City, named “AnyDay”, is taking place.
The game takes place at the beginning of autumn and starts on a Monday, 5 days before the AnyDay festival takes place, with outside activities across the river and a big concert with famous singers close to the bridge area (area C in Fig. 2a). The peak of the festival is planned for Saturday, when participants are expected to reach numbers of 10 000. Public officials are checking out the weather forecasts to ensure that Anywhere City's traditional festival can happen under the safest conditions so that participants can enjoy the next weekend camping and celebrating in the region.
Each player is given a specific sub-role to act as representative of one of the following institutions: (i) hydro-meteorological services, which interpret the hazard model outputs and communicate warnings if needed; (ii) first responder services, which deal with the possible evacuation of residences, schools, campsites and public events; (iii) the municipality, which makes decisions related to the everyday (e.g. anticipation of school pick-up time, cancellation of school-related transport) or recreational events (i.e. AnyDay festival) in the city; (iv) road services, which manage road closures and the maintenance of the main bridge road in case of a flood emergency (see worksheet for flood scenario in Appendix A). The greatest challenge is to decide whether the AnyDay festival should be cancelled or if the event could be maintained and probably set up flood protection measures in the bridge area. Withdrawing such a big event, for which people have prepared and have been looking forward to, obviously would cause upheaval and would reduce their wellbeing. In this case, the municipality would also pay cancellation fees and other expenses with the ultimate goal to prevent people from risk. On the other hand, if high water or debris flow blocks and collapses the bridge while the festival is still on, thousands of unprepared people will be exposed to severe flooding. In general, if no protective action or evacuation were decided for the day an event hit an area, the population is considered subject to risk.
3.1.2 Input data and simulations
The period of concern in flood simulations are the 5 days of the week (from Monday to Friday) preceding the day of “AnyDay festival”. The GM provides and comments on medium-range deterministic precipitation forecasts and hydrological forecasts produced by the European Centre for Medium-Range Weather Forecasts (ECMWF) for Monday and Tuesday in order to familiarize the players with the products and slowly put them in context. Each of the 3 following days represents one round of the game for which collective decisions are requested from the players, assuming that every evening the group decides on the emergency activities of the next day (Fig. 3a). In each round, the players receive area-specific information so that they can make distinct safety choices adapted to the predicted hazard in each area of Anywhere City. Low-predictability events such as flash flooding on steep slopes and domino effects like debris flow in the river are also part of the flood scenario that prompts emergent stressful situations.
Figure 3Schematic illustration of the gaming timeline (rounds 1 to 3) and the information provided to the players of ANYCaRE for the (a) flood scenario and (b) strong-wind scenario. Each of the three game rounds (R1–R3) played in the experiments corresponds to a daily or hourly time step before the festivals that, according to the storylines, are held on Saturday. In the second trial of each round, the players receive additional decision-support tools including high-resolution forecasts and impact-based vulnerability inputs.
In the first trial of each round, players receive the 24-hourly accumulated precipitation based on the ECMWF ensemble mean for Anywhere City. Precipitation is calculated as the median value in every grid cell from 51 integrations with approximately 32 km spatial resolution. Secondly, input data include hydrological forecasts for three different locations (i.e. A–C) based on the European Flood Awareness System (EFAS), which is operationally running on the European scale under the Copernicus Emergency Management Services (Copernicus EMS) (Smith et al., 2016; Thielen et al., 2008). EFAS products were particularly selected because they present successful implementation of probabilistic medium-range flood forecasts with special attention on the communication of uncertainty (Demeritt et al., 2013; Pappenberger et al., 2013) and their EU-wide operational availability (Smith et al., 2016).
The second trial proposes additional radar-based high-resolution rainfall accumulation forecasts (i.e. every 15 min with lead times up to 6 h) produced by the European Rainfall-InduCed Hazard Assessment (ERICHA), and EFAS bias-corrected forecast hydrographs where near-real-time discharge observations are used to compensate the propagation of the error between forecasted and observed discharge (Bogner and Kalas, 2008). Advanced hydro-meteorological products are selected to show the importance of end-user tailored forecast visualizations and highlight the potentiality of combining (i) continental-scale medium-range forecast at coarse spatio-temporal resolution with (ii) local, high-resolution and short-range predictions to improve timely detection and better localization of extreme events. These include return-level hydrographs (i.e. time series of the return levels of corresponding forecasted discharge values at the same time step) and exceedance plots (i.e. daily box plots corresponding to different lead times of the hydrological forecast) from EFAS forecasts driven by the ECMWF ensemble.
The second group of products presented in the second trial focuses on the importance of translating the hazard into risk information, and especially probabilistic risk assessments, for the decision-making. Impact-based products are based on the Rapid Impact Assessment layer that combines event-based hazard maps with exposure information to assess several categories of impacts such as affected population and damage (Dottori et al., 2017) (more information available at http://anywhere-h2020.eu/catalogue/?product=river-flood-impact-forecast, last access: 11 March 2019). Direct economic losses are computed by combining the “Coordination of information on the environment” (Corine) map with flood hazard variables (i.e. flood extent and depths) and a set of damage functions derived for European countries. In addition to that, the extent of urban and agricultural areas affected is computed using the CORINE Land Cover inventory (Dottori et al., 2017). The last information presented to players emphasizes the benefits of the seamless integration of the volunteer geographic information (VGI) collected during the crisis from social media and via dedicated crowdsourcing applications (i.e. artificial tweets and posts in a hypothetical crowdsourcing system drawn especially for the experiment). VGI content is widely recognized as a crucial source of information that cannot be measured directly (e.g. socio-economic impacts, human perception), and thus, it is assumed to positively contribute to the overall situational awareness and crisis decision-making.
3.2 Strong wind scenario
3.2.1 Storyline and roles
The strong-wind scenario was built based on the structure of the flood scenario with necessary adjustments. The experiment was originally designed for a combined training day of Finish civil protection services, meteorologists and electricity company employees in the South Savo municipality. Therefore, the territory and the game roles were selected as representative of the study area. The South Savo municipality has a special landscape with (i) dense forest areas, (ii) power lines above the ground that are vulnerable to strong winds and falling trees, (iii) lake areas attracting numerous boatmen during the summer, and (iv) the urban areas of Mikkeli and Juva. In the storyline, we illustrate areas A–C to represent the variety of these different areas (Fig. 2b). Area A represents the urban area of Mikkeli with a year-round population of 50 000. The game takes place in mid-July and starts on Friday, 24 h prior the Jurassic Rock festival and Sulkavan Soutu boat race. The music festival is expected to attract an additional 25 000 persons to Mikkeli (Area A) and 4000 boatmen to the boat race on the lake at Sulkava (Area B). The urban area also includes a lot of critical infrastructure like chemical factories, hospitals and shops that are dependent on a continuous electricity supply. Finally, Area C mainly contents the road leading from Juva to Sulkava, which is a crucial evacuation route for civil protection services when facing emergencies at the lake areas. Since the road tends to get blocked by falling trees in storms, it was included into the game storyline to create an additional challenge for the civil protection service's decisions.
The players of the strong-wind scenario act as (i) meteorologists, who interpret the NWP model information to the customers and issue warnings (and the corresponding meteorological bulletin); (ii) civil protection services, which make decisions for possible evacuation of public events (e.g. Jurassic Rock festival); (iii) electricity companies, which manage the maintenance of electricity distribution to the customers considering the related economic constraints (see worksheet for strong-wind scenario in Appendix A). The electricity company representatives are especially challenged to divide their resources effectively in order to fix potential power cuts as quickly as possible and ensure the electricity supply in the area. Similarly to the flood scenario, the biggest challenge for the players refers to the protection of the festival planned for Saturday.
3.2.2 Input data and simulations
The timeline in the strong-wind simulations is 24 h before the public events. Given the rapid development tendency, the small size and the short life cycle of convective storms, the forecast time is among the shortest lead times in weather phenomena, making forecasting a real challenge. Therefore, each round is chosen to be in terms of hours prior to the event (i.e. round 1 – 24 h, round 2 – 6 h, round 3 – 1 h ahead) in the ANYCaRE game (Fig. 3b). From meteorological perspective, the scenario imitates the so-called “Asta” strong-wind event that happened in 2010 and caused financial losses of over EUR 20 million (Astola et al., 2014). This particular event was selected because most of the trainees for whom the experiment was designed had indeed experienced strong Asta winds. Living through the event again (but without knowing it at the beginning of the game) and facing it under different circumstances, the players were offered the possibility to compare the existing tools to the new impact tools and to realize the possible consequences of such a strong-wind event in different situations (e.g. occurrence during daytime on the busiest summer weekend compared to the original event that took place during night).
The simulations replicate the KRIVAT briefing. KRIVAT is a platform supporting collaboration between enterprises managing critical infrastructure and authorities managing major disruptions to services (more information available at https://www.erillisverkot.fi/en/services, last access: 11 March 2019). During KRIVAT briefings different actors held a teleconference on Fridays to receive a weather outlook for the weekend. In the first round (24 h prior to the event), the GM provides information about the synoptic situation and the meteorological forecast of ECMWF and the Global Forecast System (GFS) in the area. The scenario also presents imaginary but realistic cascading effects and unpredictable events such as falling trees over the power lines and, subsequently, power outages in critical areas (e.g. close to the chemical factory). In such urgent situations, the civil protection service needs to react fast to cooperate and communicate with the meteorologists and the electricity company's staff. In each round, existing tools with only weather information are presented in the first trial, whereas the second trial provides additional impact-based tools. In particular, in the second trial, meteorologists receive model calculations outputs from the “Expert System for Consequence Analysis and Preparing for Emergencies” (ESCAPE) model calculations and interpret them for civil protection services. The ESCAPE model has been designed for evaluating the dispersion of toxic gases into the atmosphere and the effects on humans and the environment (Kukkonen et al., 2017). Second-trial tools emphasize the identification of the severity of the convective cells, their detailed route and the economic impact estimation that is particularly interesting to the electricity company. A common approach for the automatic identification and short-term forecasting of thunderstorms is the object-oriented storm tracking, which follows the movement of individual storms from remote-sensing data and extrapolates the storms based on the tracking information (Rossi et al., 2013). Severity categories of convective cells are classified by using failure data from the electricity company and comprise interesting input for forecasters and civil protection services. This information is assumed to also facilitate the optimization of electricity grid actions undertaken by the electricity company. Also, in the second trial, the civil protection service has access to the A4FINN platform and the available impact map. A4FINN, a product developed for the municipality of South Savo by the ANYWHERE H2020 project, combines weather model outputs and data from different sources with impact estimations. For instance, the emergency response plans of the civil protection services are now connected with weather information, which brings the experienced and less experienced emergency responders to the same level when making decisions. The platform provides a tool indicating the weather situation severity level in different municipalities. A4FINN includes relevant weather parameters and a map product, which shows the automatic, suggested preparedness level of the civil protection services.
Figure 4From (a) to (d): players debate during the role-playing simulations of ANYCaRE in (a) the Helsinki experiment (21 September 2017), (b) the Grenoble experiment (24 January 2018), (c) the UGA experiment (11 April 2018) and (d) the Mikkeli experiment (17 May 2018). (e) Example of players' post-experiment debriefing post-it notes (photo taken in the Helsinki experiment).
The first ANYCaRE experiment was carried out in the frame of the ANYWHERE European project workshop in Helsinki (Finland) on September 2017. This workshop gathered 161 attendees including partners of the project and external organizations involving researchers, forecasters, civil protection services and representatives of related companies in Europe. A game session was organized with a group of sixteen players to compose the virtual EOC of Anywhere City and play the flood scenario (Fig. 4a). The main purpose of the experiment was to check the validity of the scenario and test the functionality of the game with experts from different fields and agencies across Europe (Table 3). Another flood game testing was organized 4 months later in a European scientific meeting in Grenoble (France) which had the similar goal of confirming improvements and enhancing the game-mastering experience before further application. Among participants in the two test experiments there were PhD students and researchers in weather-related hazards, developers and modellers, emergency managers and operational forecasters (Fig. 4b). Therefore, player experiences in weather-induced risk modelling and management ranged from low (i.e. <5 years) or no operational experience to high (i.e. >10 years), providing genuine diversity to be reflected in the games.
Table 3Information for the game sessions conducted within ANYCaRE.
After the simulations, the game observers invited the players to provide feedback on both the experiment set-up and the input data presented in the game. The test experiments were considered successful since the game was found “to be representative of the reality of flood crisis management” by players who had the experience of such a situation and “clearly demonstrated the benefit of certain products”. In both the Helsinki and Grenoble experiments, players agreed that the simulations were motivating and “kept the stress”, and the whole set-up was a “very representative group exercise”. That made participants appreciate the game and recommend it for play in future events. Players introduced further potentials for ANYCaRE, emphasizing the following:
-
Training such as coaching emergency services sharpens their emergency agility and alertness before the crisis strikes: “The game can be used in my organization to test the emergency response” (Helsinki player); “I would like to use the game as part of the training in the hospital emergency dissipation. Good way to improve their way of taking decisions” (Grenoble player).
-
The educational context was useful: “It is a very useful exercise and I would love for it to become also a tool to educate different segments of the population” (Grenoble player).
Considering such potentialities, two more experiments were organized in spring 2018 as presented in Table 4. In the first experiment, 10 undergraduate students (3rd year of a Bachelor of Technology programme) of the University of Grenoble Alpes (UGA) played ANYCaRE flood simulations as part of their course (Fig. 4c). Students following this scientific and professional training are especially prepared for a practical integration into the operational world of water resources and risk management. The benefit of this gaming practice was evaluated through a short questionnaire answered both before and after gameplay.
The second experiment was integrated into the official annual training of the Finnish stakeholders involved in the management of hazardous weather emergencies (i.e. meteorological services/FMI, civil protection services in the Eastern Finland region, electricity providers from “Järvi-Suomen Energia”), which was held on May 2018 in Mikkeli (Finland) (Fig. 4d). Impact products developed by the Finnish Meteorological Institute (FMI) were introduced to the trainees with the aim of familiarizing end users with new developments, introducing them to the products' beneficial use as operational decision-support tools and rehearsing their communication for an effective response during weather hazards like strong winds. A post-experiment questionnaire was also designed, in particular to assess whether the game met the learning objectives as expected.
Based on improvements proposed by participants in the test experiments, the game designers considered various ways to ease the gaming process in the two new game sittings such as (i) documenting and distributing the input data for every role in separate booklets, (ii) supplementing the initial worksheet with more specific assignments for certain roles (e.g. road closing to be decided by the road services representatives in the flood scenario), (iii) distributing demanding roles according to the real-life competencies and players' backgrounds (e.g. give forecasting responsibilities to players with forecasting experience).
Table 4Themes of the players' comments recorded in the post-it notes in the debriefing phase of the four ANYCaRE experiments. Every comment was placed in one theme. Post-it notes with more than one thematic annotation were assigned to more themes to match comments to theme.
The four experiments included a total of 46 players. All of them orally participated in the debriefing phase and optionally quickly recorded their main ideas on paper. Participants either commented on positive remarks or suggestions for improvements related to the game itself but also to the modern products provided as input data to support decision-making in the simulations. Short textual annotations on post-it notes were gathered and placed in the corresponding category on a board to open the discussion (Fig. 4e). Table 4 presents a summary of the themes in which the feedbacks are classified after qualitative analysis of the players' written comments (Rebolledo-Mendez et al., 2009). In the third experiment, the students replied also to short pre- and post-experiment questionnaires. Similarly to previous studies on role-playing gaming (Huyakorn et al., 2012), the questionnaires comprised the same open questions to identify differences (if any) in their perceptions on risk management after playing the ANYCaRE game. Trainees of the fourth experiment had the opportunity to also provide detailed feedback on the gaming experience and the related learning outcomes by answering Likert-scale questions as well as complementary open questions supplementing their statements.
5.1 Feedbacks from observations of the gameplay and debriefing discussions
5.1.1 ANYCaRE is a valuable communication and learning tool
It is important for participants to feel that the game experience was playable, the rules were understandable and the learning procedure was attractive (Dieleman and Huisingh, 2006; Turkay and Adinolf, 2012). After playing the ANYCaRE game, the majority of participants in the four conducted experiments expressed their satisfaction for the gaming experience. Based on the categorization of the comments provided on post-it notes during the debriefing sessions, we found that players noted their appreciation for the “stimulating” (six instances), “very fun” (four instances) learning approach experienced in ANYCaRE (see Table 4 for details of all comments and related instances). It was mentioned that it was “very interesting to try roles different from the usual day-by-day experience” (Helsinki player). The experiments were found to fulfil the main hypothesis, that the game offers a safe environment in which to enact different personas and gain a deeper understanding of the decision-making challenges met by the involved actors (eight instances): “The game is a good way to start handling emergency management with a different approach. Also, people that are not in their own role can understand the struggles of the others” (Grenoble player); “New experience that helps to understand better other roles” (Mikkeli player); “Good way to put you in the shoes of actors” (UGA player).
In all the experiments, the players recognized ANYCaRE scenarios as very realistic and presented a strong commitment to the storytelling (six instances): “Very real feeling and I really took the game seriously” (Grenoble player); “Very good game and incredibly realistic!” (Mikkeli player). During the feedback sessions in the Helsinki and Grenoble experiments as well as in the Mikkeli training, ANYCaRE players discussed that role playing was a good approach to bringing people from different institutions together to share knowledge and experience while examining new (pre-) operational tools and their potentialities. They often expressed their preference on ANYCaRE's role-playing structure for training decision-making skills on emergency management (four instances): “Would use this exercise in our later trainings for civil protection” (Mikkeli player). On the other hand, UGA students appreciated the game for educating them on their future responsibilities: “Good exercise/learning for the following because it gives concrete example”.
5.1.2 Modern impact-based information increase the level of confidence in emergency management decisions
At the beginning of each round, the GMs described the existing weather-related conditions and explained if and why the group decision taken in the previous round was relevant or not to ensure safety with the minimum losses. It appeared that the players in the test experiments as well as in Mikkeli training already did a relatively good job in Trial 1 in the first round (e.g. by evacuating scouts from the campsite before the occurrence of a severe flash flood in area A in the flood scenario). Some players in the Grenoble experiment mentioned that improved forecasts including probabilistic information for exceeding return levels were very informative (four instances). Though, in some experiments it was observed that advanced meteorological forecasts such as high-resolution precipitation maps and flood probabilistic forecasts provided in Trial 2 did not necessarily lead to better decisions for emergency response. Still, this result may be subject to specific skills of the players who acted as forecasters as well as the overall experience of the participants in these sessions. For instance, operational meteorologists participating in Mikkeli training were able to detect significant signs about a potentially severe strong-wind situation already from the first round, even with a limited amount of information. Thus, they gave a very good briefing to the others, creating a confident atmosphere in which other actors could select emergency actions. This was not the case when the much less experienced UGA students played within the flood simulations. Although differences between Trial 1 and Trial 2 were not always obvious in terms of the selected emergency activity on the worksheets, in most of the cases, players in the four experiments rated their confidence more highly when they passed on to Trial 2.
In the debriefing, some players of the flood scenario simulations mentioned that detailed meteorological data are not trivial to non-forecasting-specialists (four instances): “Difficult information for operators in emergency centres” (Helsinki player); “For a non-expert, it is difficult to understand the probabilistic info provided by forecasting systems” (Grenoble player). Yet, hydro-meteorological forecasting was observed to dominate the decision-making and related discussion compared to impact estimations in the first two experiments. The limited use of the new impact-based tools in the game may be attributed to the lack of players' familiarity with the new products opposed to previously seen operational tools. Other potential explanations that require further examination may include (i) inadequate understanding on how to handle the new impact information during crisis, (ii) absence of trust in the new developments and (iii) incapability of the adopted visualizations to convey helpful information. In UGA experiment, where students were much less experienced with the provided hydro-meteorological products, decisions between trials 1 and 2 changed greatly considering the impact-based vulnerability information and, especially, the social media posts.
In the debriefing discussions, the players agreed that the information provided, in particular from impact estimations and social media, was very useful for better targeting their actions. Generally, in all the experiments players seemed to largely rely on impact observations assumed as reported through comments and pictures on social media. This was particularly the case in the third and last rounds of the three flood-scenario experiments in which the players completely changed their emergency decision after receiving a crowdsourced image showing the bridge blockage with wood and debris. The payers shifted from “no action” to the set-up of flood protection measures in area C, and finally, the closure and maintenance of the bridge road in Trial 2. The examples of modern impact-based vulnerability information included in the experiments were found to reduce the overall uncertainty in the decision-making process. The multi-model outputs were characterized as useful, especially when accessible to multiple actors directly without the need from mediation from other (governmental) agencies (three instances): “The ability to offer and share valuable data or information directly to consumer is brilliant. It avoids putting someone's safety in the hands of government decision makers” (Grenoble player). Expert forecasters and members of the civil protection service in Mikkeli experiment found that the variety of information presented through the A4FINN tool and impact-based maps helped them to shape a holistic view of the situation and increased their confidence in decision-making. The players of all experiments acknowledged that the variety of new products enhanced their sureness about choosing specific emergency activities and communicated in particular areas.
5.2 Analysis of the post-experiment questionnaires responses
In the Grenoble experiment with UGA students, the 10 game participants were asked the same three questions before and after playing the game to evaluate the potential change in their perception related to weather-related crisis management. The open questions were the following:
-
What key words do the terms “crisis management” evoke for you?
-
What types of information are necessary to take a decision with respect to an imminent inundation threat?
-
According to you, what challenges do crisis decision makers have to face when flooding occurs?
The qualitative analysis of the differences between the answers pre- and post game indicates that students did not imagine how much knowledge sharing, communication and coordination effort was needed for crisis management. “Anticipation”, “information”, “communication” and “actors” are key words that were not cited before the game but were cited several times after the game. With respect to the second question related to the type of information necessary for crisis decision-making, playing the game did also make a difference. Students realized that not only were historical data about peak discharge, damage or the identification of elements at stake useful but so were weather prediction, vulnerability and accessibility data, as well as more contextual information (like the ones concerning the occurrence of big events potentially increasing temporarily the exposed population). Finally, in terms of the perception of the challenges faced by crisis decision makers, the players highlighted the difficulty of managing the multitude of data, prioritizing actions in order to better anticipate and making the optimal decisions without being overwhelmed. Key words like “optimal decision/management”, “elements at stake” and “anticipation” were mostly cited after playing the game.
After Mikkeli experiment, 5 of the 12 participants also provided written responses to a questionnaire, which was targeted to evaluate the usefulness of the ANYCaRE gaming simulations in the training process. This low response rate could be attributed to the fact that everyone first participated in the oral debriefing session and did not feel the need to add any more comments. The questionnaire asked players to rate their satisfaction in terms of training experience and applicability of their learning on a five-level Likert scale. In agreement with the oral debriefing discussion, most of the players who answered the questionnaire (3 out of 5) noted that ANYCaRE fairly fulfilled their expectations, highlighting that the simulations promote co-operation and communication between crucial actors toward the formation of common situational awareness and preparedness strategies. According to the players, forming the complete situational picture is one of the main challenges in weather crisis management and it was also depicted as a challenge while playing. Players evaluated the learning outcomes of ANYCaRE though various questions. Nearly all respondents stated that they learned a lot through playing. All of the respondents highly rated their ability to apply this learning in their professional environment and they largely proposed the gaming activity as a relevant training tool. In summary, the respondents indicated the following take-away messages:
-
The training through the game was very pleasant. The process revealed the importance of certain roles in the decision-making chain and taught the significance of co-operation between multiple actors for efficient problem solving.
-
The lessons can be immediately applied in the corresponding agencies, increasing preparedness for different weather-related situations. The simulations help participants to get an overview of the multi-faceted demands in emergency response and to identify products that can be used for the anticipation of impacts in affected areas in the future.
-
The experiment proposes a flexible framework that can be easily adapted to host the needs and purposes of different training activities (e.g. training of the municipality lead group to weather-related hazardous events). With certain modifications, ANYCaRE can be extended to other tabletop exercises or to be digitalized to offer a modern multi-task and multi-role structure for a deeper understanding of decision makers' challenges in weather crises.
5.3 ANYCaRe's limitations
The biggest challenge identified in the role-playing exercise is to engage players equally and empower their active presence in the decision-making. In our experiments and similarly in previous game studies, the personality of the player was observed to be an important factor during the game, with the more extrovert and talkative players dominating the decision-making and the shyer ones participating much less in the debate (Mossoux et al., 2016). Shy players were involved in the argumentation when they felt the need to defend a specific priority related to their role. In addition to that, players already experienced with certain processes and/or tools (e.g. experienced forecasters in Mikklei training) tended to largely rely on their prior expertise when leading decisions in the game. This may hinder the reflection of certain facts tested in the experiment (e.g. confidence on the decision-making based on the presented information). To facilitate the representation and relevant intervention of all the roles and their particularities in the debate, it was suggested that very detailed responsibilities and constraints should be assigned to each role at the beginning of the game; allowing the players to set up a common strategy with others playing the same role (eight instances). Smaller groups with up to 12 players were also perceived as more controllable and efficient to prompt player's engagement in the game world.
The gaming process was diagnosed as having also limitations related to the time needed for the players to understand the concept and get ready to play (four instances) as well as processing of complex hydro-meteorological information probably unfamiliar to some participants (seven instances). Although the current level of complexity in ANYCaRE made the game to be stimulating for the players, it was observed that the given hydro-meteorological information was sometimes difficult to digest in a short time for participants without forecasting expertise. Obviously, more time should be given in the presentation and explanation of the elaborated data before the simulation starts. Another perspective is to introduce level 1 simulations in which the forecasters' group will work separately to prepare and deliver forecasts to the emergency group (five instances).
Timing is overall an important issue, requiring a balance between having adequate time for debate and decision-making and representing realistic stressful crisis situations. Although the time available for taking and recording decisions on the worksheets was increased after some propositions made in the test experiments, players in the Mikkeli and UGA experiments still found the 10 mins very limited, especially in the first round. However, designers should consider that, as the total time required for the experiment increases, the applicability of the game in different settings (e.g. workshops, short training sessions) is hindered. A well-trained games master is definitely necessary to clearly introduce material appropriate to the players and guide them through the game, optimizing the total time required for the experiment.
This paper presents a role-playing experiment designed to investigate crisis decision-making in weather-related risks. ANYCaRE allows us to explore how decision makers and stakeholders interact with scientific and operational outputs to better anticipate and respond to extreme and high-impact weather and climate events in Europe. The experiment includes tabletop gaming simulations on common hazards under consideration in European sites such as (i) flooding and flash flooding, and (ii) strong winds and thunderstorms. From September 2017 to May 2018 flood-based ANYCaRE was applied in two European experiments and one educational experiment in a French university, whereas the strong-wind scenario was played for training purposes with Finish operational forecasters, emergency managers and stakeholders.
First results show that the ANYCaRE game aroused the interest and enthusiasm of participants and offered to the players a protected environment to try out emergency actions without facing true risks for human life. In both scenarios, the players affirmed that the simulations adequately reflected situations found in the real world and facilitated their involvement in the storytelling. It was observed that this participatory technique set a playful and collaborative atmosphere between scientific partners and stakeholders, generating fruitful debate on appropriate emergency decisions. Players highlighted the value of communication between the involved actors, which was successfully represented in the game. Compared to conventional trainings, the gaming approach presents the possibility of introducing more than one tool at a time to the end users, generating cross-interest in each other's decision-support tools and related challenges in the decision-making. Therefore, the gaming simulations sufficiently served the educational and training goals of the experiments.
As a proof of concept, this first study did not include yet an explicit evaluation of the products presented as inputs and their contribution to the decision-making. At the final debriefing step of the experiments, participants were rather encouraged to exchange knowledge, thoughts and insights into their capabilities or difficulties in deciding and communicating their action based on the available information and given constraints. In a broad view, in all the experiments the players complied well with the scenario requests. The main conclusion drawn from the first applications is that modern multi-model outputs provided extra confidence in the decisions taken in the first trials based solely on existing hydro-meteorological information. Expert forecasters and members of the civil protection service in Mikkeli experiment found that the variety of information presented through impact-based tools helped them to shape a holistic view of the situation and increased their confidence in decision-making. However, this fact was not necessarily obvious when examining the group responses on the worksheets. It appears that decision-making is largely influenced by personal attributes such as prior experience on weather forecasting and familiarity with disastrous events. Concerning the vulnerability information, all the players of the ANYCaRE game agreed that online crowdsourcing tools might be a great provider of ground facts necessary to enhance situational awareness of authorities, especially in cases of high hydro-meteorological uncertainties and forecasting failures. Future developments were suggested to geolocate the social media content to help emergency responders to clearly identify places where urgent action is needed. Nevertheless, interviews conducted with emergency communication experts in the context of the ANYWHERE project also show that the reliability of such crowdsourcing information is a concern and there is a necessity to check the information in real time and potentially counter false rumours as soon as they emerge (Müller et al., 2017).
Communication to the public was only a small part of the players' duties in ANYCaRE's first implementations. To simplify the game the players did not have to compose an emergency message by themselves but had some simplified examples to choose from. Therefore, choosing the relevant official emergency response and deciding on whether that should be communicated to the population in specific locations of the study area was considered the main responsibility of the emergency group. When the hypothetical crowdsourcing system indicated the bridge blockage in Anywhere City, for example, the emergency group of flood experiments decided to inform the public immediately of the imminent risk. The players opted for specific guidance, indicating that the public should stay away from the bridge and the festival area. Other communication aspects could be addressed in the future by asking players to choose a specific format and delivery method for the conveyed information. That would enable us to examine what their criteria are for the distribution of understandable and usable messages to the public.
Rather than a single tabletop role-playing game, we vision ANYCaRE as a broad experiment campaign that will encompass various versions of games to be applied in different settings. Applications to other weather-induced risks such as wildfires were also encouraged by participants in the past experiments and are discussed for implementation. A series of future expansions is considered to (i) adjust scenarios to other weather hazards including multi-hazard cases and complex cascading effects commonly challenging European cities, (ii) test additional models and technological innovations (e.g. crowdsourcing tools, dialogue systems, internet-based apps) and (iii) establish other formats of serious gaming such as online or board games to attract different audiences (e.g. stakeholders, general public, pupils) and, subsequently, enlarge the amount and variety of feedbacks on weather crisis decision-making. Online playing would allow the player to experience multiple simulations until they establish a deeper understanding of relevant interpretations of the new information and successful paradigms for emergency responses. Such developments will offer a floor for testing hypotheses on the usefulness of new products and their reception by experts or the public. ANYCaRE, whether digitalized or played online, could be used as a communication tool between developers of new decision-support tools and end users to provide a feedback loop for further improvements to developers. More targeted questions could be examined: “Is the product user-friendly and easily understood?”; “Does it make decision-making easier for the users?”; “If yes, what is the characteristic that makes it useful (e.g. the spatial or temporal specificity, precision)?”; “If not, then what needs to be improved?”. When the game refers to the general public, more detailed questions dedicated to risk communication could be addressed: “Does the population at risk receive the risk/warning information and emergency messages through the selected mean(s)?”; “Are the timing and geographic specificity of the messages appropriate to interrupt the public's daily routine activities in favour of crisis-related preparedness and protection?”. Our ultimate purpose is to draw more detailed conclusions on the effectiveness of impact-based forecast visualizations and delivered warning and emergency messages (i.e. content, structure and format) in terms of comprehension and mobilization of action. Such knowledge is a prerequisite for the anticipation of effective crisis communication strategies and relevant emergency responses to prevailing weather threats.
The qualitative data used to assess the game are all summarized in Table 4.
Table A1Worksheet with predefined list of emergency and communication activities delivered to players of the flood scenario.
Table A2Worksheet with predefined list of emergency and communication activities delivered to players of the wind scenario.
The order of the authors' names reflects the size of their contribution to the writing of this paper.
The authors declare that they have no conflict of interest.
This project has received funding from the European Union's Horizon 2020
research and innovation programme (H2020-DRS-1-2015) under grant agreement no. 700099.
Edited by: Sven Fuchs
Reviewed by: Giovanna Lucia Piangiamore and one anonymous referee
Abt, C. C.: Serious Games, Viking Press, New York, 1970.
Alharthi, S. A., LaLone, N., Khalaf, A. S., Torres, R. C., Nacke, L. E., Dolgov, I., and Toups, Z. O.: Practical insights into the design of future disaster response training simulations, in: Proc. 15th ISCRAM Conference, May 2018, Rochester, NY, USA, 818–830, 2018.
Astola, H., Molinier, M., Simons, M., and Susila, P.: Forest stem volume and storm damage mapping in Finland and Russia, in: IEEE International Geoscience and Remote Sensing Symposium (IGARSS), 13–18 July 2014, Quebec City, Canada, 2309–2312, 2014.
Aylett, R., Louchart, S., Tychsen, A., Hitchens, M., Figueiredo, R., and Mata, C. D.: Managing Emergent Character-Based Narrative, in: Proc. 2nd Int. Conf. Intell. Technol. Interact. Entertain., 8–10 January 2008, Cancun, Mexico, https://doi.org/10.4108/ICST.INTETAIN2008.2468, 2008.
Backlund, P. and Hendrix, M.: Educational games – Are they worth the effort? A literature survey of the effectiveness of serious games, in: IEEE 2013 5th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES), 11–13 September 2013, Poole, UK, 1–8, 2013.
Bogner, K. and Kalas, M.: Error-correction methods and evaluation of an ensemble based hydrological forecasting system for the Upper Danube catchment, Atmos. Sci. Lett., 9, 95–102, https://doi.org/10.1002/asl.180, 2008.
Bogost, I.: Persuasive Games. The Expressive Power of Videogames, Mit Press, Cambridge, Massachusetts, 2007.
Bowman, S. L.: The functions of role-playing games: how participants create community, solve problems and explore identity, editedb by: Jefferson, N. C., McFarland & Co., Jefferson, North Carolina, USA, 2010.
Boyle, E. A., Macarthur, E. W., Connolly, T. M., Hainey, T., Manea, M., Kärki, A., and Van Rosmalen, P.: A narrative literature review of games, animations and simulations to teach research methods and statistics, Comput. Educ., 74, 1–14, https://doi.org/10.1016/j.compedu.2014.01.004, 2014.
Cover, J. A. G.: Tabletop Role-Playing Games: Perspectives from Narrative, Game, and Rhetorical Theory, Graduate Faculty of North Carolina State University, Raleigh, 2005.
Cover, J. A. G.: The creation of narrative in tabletop role-playing games, McFarland & Company, Jefferson, North Carolina, USA, 2010.
Creutin, J. D., Borga, M., Gruntfest, E., Lutoff, C., Zoccatelli, D., and Ruin, I.: A space and time framework for analyzing human anticipation of flash floods, J. Hydrol., 482, 14–24, https://doi.org/10.1016/j.jhydrol.2012.11.009, 2013.
Crichton, M. T., Flin, R., and Rattray, W. a. R.: Training Decision Makers – Tactical Decision Games, J. Conting. Cris. Manage., 8, 208–217, https://doi.org/10.1111/1468-5973.00141, 2000.
Demeritt, D., Nobert, S., Cloke, H. L., and Pappenberger, F.: The European Flood Alert System and the communication, perception, and use of ensemble predictions for operational flood risk management, Hydrol. Process., 27, 147–157, https://doi.org/10.1002/hyp.9419, 2013.
Dieleman, H. and Huisingh, D.: Games by which to learn and teach about sustainable development: exploring the relevance of games and experiential learning for sustainability, J. Clean. Prod., 14, 837–847, https://doi.org/10.1016/j.jclepro.2005.11.031, 2006.
Di Loreto, I., Mora, S., and Divitini, M.: Collaborative serious games for crisis management: an overview, in: 2012 IEEE 21st International Workshop on Enabling Technologies: Infrastructure for Collaborative Enterprises, 25–27 June 2012, Washington, D.C., USA, 352–357, 2012.
Djaouti, D., Alvarez, J., and Jessel, J. P.: Classifying serious games: the G/P/S model, in: Handbook of research on improving learning and motivation through educational games: Multidisciplinary approaches, IGI Global, Hershey, PA, USA, 118–136, 2011.
Dottori, F., Kalas, M., Salamon, P., Bianchi, A., Alfieri, L., and Feyen, L.: An operational procedure for rapid flood risk assessment in Europe, Nat. Hazards Earth Syst. Sci., 17, 1111–1126, https://doi.org/10.5194/nhess-17-1111-2017, 2017.
Drachen, A., Copier, M., Hitchens, M., Montola, M., Eladhari, M. P., and Stenros, J.: Role-Playing Games: The State of Knowledge, in: Breaking New Ground: Innovation in Games, Play, Practice and Theory, in: vol. 6, Proceedings of DiGRA 2009, September 2009, Brunel University, Brunel, p. 13, 2009.
Friman, R.: The Crisis Game, Simul. Gaming, 22, 382–388, 1991.
Garcia Londoño, C. and Fearnley, C.: Adapted Technologies for Early Warning Systems: Playing with Uncertainty, in: Proc. EPFL-UNESCO Tech4Dev 2018: Voices of the Global South, 27–29 June 2018, Lausanne, Switzerland, 2018.
Heinsoo, R., Collins, A., and Wyatt, J.: Dungeons & Dragons Player's Handbook: Arcane, Divine, and Martial Heroes (Roleplaying Game Core Rules), Wizards of the Coast, Renton, WA, USA, 2008.
Hémon, D. and Jougla, E.: The heat wave in France in August 2003, Epidemiol. Public Heal. d'Epidémiologie Santé Publique, 52, 3–5, 2004.
Huang, C.-C., Yeh, T.-K., Li, T.-Y., and Chang, C.-Y.: The Idea Storming Cube: Evaluating the Effects of Using Game and Computer Agent to Support Divergent Thinking, Educ. Technol. Soc., 13, 180–191, 2010.
Huyakorn, P., Denpaiboon, C., Kanegae, H., and Management, A. C. F.: Role-play Gaming Simulation for Flood Management on Cultural Heritage: A Case Study of Ayutthaya Historic City, World Acad. Sci., 6, 111–116, 2012.
Kolb, D. A.: Experiential Learning: Experience as The Source of Learning and Development, Prentice Hall, Inc., Englewood Cliffs, NJ, 20–38, https://doi.org/10.1016/B978-0-7506-7223-8.50017-4, 1984.
Kox, T., Lüder, C., and Gerhold, L.: Anticipation and Response: Emergency Services in Severe Weather Situations in Germany, Int. J. Disast. Risk Sci., 9, 116–128, https://doi.org/10.1007/s13753-018-0163-z, 2018.
Kukkonen, J., Nikmo, J., and Riikonen, K.: An improved version of the consequence analysis model for chemical emergencies, ESCAPE, Atmos. Environ., 150, 198–209, https://doi.org/10.1016/j.atmosenv.2016.11.050, 2017.
Lee, Y. I., Trim, P., Upton, J., and Upton, D.: Large emergency-response exercises: Qualitative characteristics – A survey, Simul. Gaming, 40, 726–751, 2009.
Linehan, C., Lawson, S., and Doughty, M.: Tabletop Prototyping of Serious Games for `Soft Skills' Training, in: IEEE 2009 Conference in Games and Virtual Worlds for Serious Applications, 23–24 March 2009, Coventry, UK, 182–185, 2009a.
Linehan, C., Lawson, S., Doughty, M., Kirman, B., Pool, B., and Ln, U. K.: There's no `I' in `Emergency Management Team': Designing and evaluating a serious game for training emergency managers in group decision making skills, in: Proceedings of the 39th Conference of the Society for the Advancement of Games & Simulations in Education and Training, Innovation North-Leeds Metropolitan University, Leeds, 20–27, 2009b.
Luther, J., Hainsworth, A., Tang, X., Harding, J., Torres, J., and Fanchiotti, M.: World Meteorological Organization (WMO) – Concerted International Efforts for Advancing Multi-hazard Early Warning Systems, in: WLF 2017: Advancing Culture of Living with Landslides, edited by: Sassa, K., Mikoš, M., and Yin, Y., Springer, Cham, 129–141, 2017.
Mayer, I. S.: The gaming of policy and the politics of gaming: A review, Simul. Gaming, 40, 825–862, 2009.
Mayer, R., Sprenger, R., Tait, M., Macfarlane, K., and Wyse, P.: Portugal's biggest wildfire: `We all thought we were going to die' – video, Guard, available at: https://www.theguardian.com/world/video/2017/sep/19/, last access: 30 September 2017.
Meesters, K. and Van De Walle, B.: Disaster in my backyard: A serious game introduction to disaster information management, in: ISCRAM 2013 Conference Proceedings – 10th International Conference on Information Systems for Crisis Response and Management, May 2013, Baden-Baden, Germany, 145–150, available at: http://www.scopus.com/inward/record.url?eid=2-s2.0-84905667190&partnerID=40&md5=131a291cf538fcc5dd82d813e39a85fe (last access: 11 March 2019), 2013.
Metello, M. G., Casanova, M. A., and de Carvalho, M. T. M.: Using Serious Game Techniques to Simulate Emergency Situations, GeoInfo, 28, 121–182, 2008.
Mossoux, S., Delcamp, A., Poppe, S., Michellier, C., Canters, F., and Kervyn, M.: Hazagora: Will you survive the next disaster?-A serious game to raise awareness about geohazards and disaster risk reduction, Nat. Hazards Earth Syst. Sci., 16, 135–147, https://doi.org/10.5194/nhess-16-135-2016, 2016.
Müller, A., Gebhardt, O., Kuhlicke, C., Capone, F., Terti, G., Ruin, I., and Gebhardt, O.: Deliverable 1.2: Report on needs and requirements from the users including advises on how to address ethical, legal and environmental issues, available at: http://www.anywhere-h2020.eu/ (last access: 11 March 2019), 2017.
Oulhaci M. A., Tranvouez, E., Fournier, S., and Espinasse, B.: A multi-agent system for learner assessment in serious games: Application to learning processes in crisis management, in: IEEE 7th International Conference on Research Challenges in Information Science (RCIS), 29–31 May 2013, Paris, France, 1–12, 2013.
Pappenberger, F., Stephens, E., Thielen, J., Salamon, P., Demeritt, D., van Andel, S. J., Wetterhall, F., and Alfieri, L.: Visualizing probabilistic flood forecast information: Expert preferences and perceptions of best practice in uncertainty communication, Hydrol. Process., 27, 132–146, https://doi.org/10.1002/hyp.9253, 2013.
Pasin, F. and Giroux, H.: The impact of a simulation game on operations management education, Comput. Educ., 57, 1240–1254, https://doi.org/10.1016/j.compedu.2010.12.006, 2011.
Pereira, G., Prada, R., and Paiva, A.: Disaster Prevention Social Awareness: The Stop Disasters! Case Study, in: IEEE 6th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES), 9–12 September 2014, Valletta, Malta, 1–8, 2014.
Petrucci, O., Papagiannaki, K., Aceto, L., Boissier, L., Kotroni, V., Grimalt, M., Llasat, M. C., Llasat-Botija, M., Rosselló, J., Pasqua, A. A., and Vinet, F.: MEFF: The database of MEditerranean Flood Fatalities (1980 to 2015), J. Flood Risk Manage., e12461, https://doi.org/10.1111/jfr3.12461, 2018.
Poussin, J. K., Wouter Botzen, W. J., and Aerts, J. C. J. H.: Effectiveness of flood damage mitigation measures: Empirical evidence from French flood disasters, Global Environ. Change, 31, 74–84, https://doi.org/10.1016/j.gloenvcha.2014.12.007, 2015.
Rebolledo-Mendez, G., Acramides, K., Freitas, S. D. E., and Memarzia, K.: Societal impact of a serious game on raising public awareness, in: Proceedings of the 2009 ACM SIGGRAPH Symposium on Video Games, Sandbox'09, New Orleans, Louisiana, 15–22, 2009.
Robert, B. and Lajtha, C.: A new approach to crisis management, J. Contingen. Crisis Manage., 10, 181–191, 2002.
Rossi, P. J., Hasu, V., Halmevaara, K., Mäkelä, A., Koistinen, J., and Pohjola, H.: Real-time hazard approximation of long-lasting convective storms using emergency data, J. Atmos. Ocean Tech., 30, 538–555, https://doi.org/10.1175/JTECH-D-11-00106.1, 2013.
Ruin, I.: Conduite A contre-courant. Les pratiques de mobilité dans le Gard: facteur de vulnérabilité aux crues rapides, available at: http://tel.archives-ouvertes.fr/tel-00258018/fr/ (last access: 11 March 2019), 2007.
Ruin, I., Creutin, J. D., Anquetin, S., and Lutoff, C.: Human exposure to flash floods – Relation between flood parameters and human vulnerability during a storm of September 2002 in Southern France, J. Hydrol., 361, 199–213, https://doi.org/10.1016/j.jhydrol.2008.07.044, 2008.
Ruin, I., Lutoff, C., Boudevillain, B., Creutin, J.-D., Anquetin, S., Rojo, M. B., Boissier, L., Bonnifait, L., Borga, M., Colbeau-Justin, L., Creton-Cazanave, L., Delrieu, G., Douvinet, J., Gaume, E., Gruntfest, E., Naulin, J.-P., Payrastre, O., and Vannier, O.: Social and Hydrological Responses to Extreme Precipitations: An Interdisciplinary Strategy for Postflood Investigation, Weather Clim. Soc., 6, 135–153, https://doi.org/10.1175/WCAS-D-13-00009.1, 2014.
Salas, E., Wildman, J. L., and Piccolo, R. F.: Using Simulation-Based Training to Enhance Management Education, Acad. Manage. Learn. Educ., 8, 559–573, 2009.
Smith, P., Pappenberger, F., Wetterhall, F., J, T., Krzeminski, B., Salamon, P., Muraro, D., Kalas, M., and Baugh, C.: On the operational implementation of the European Flood Awareness System (EFAS), in: Flood Forecasting, edited by: Adams, T. E. and Pagano, T. C., Academic Press, New York, 313–348, 2016., 2016b.
Solinska-Nowak, A., Magnuszewski, P., Curl, M., French, A., Keating, A., Mochizuki, J., Liu, W., Mechler, R., et al.: An overview of serious games for disaster risk management – Prospects and limitations for informing actions to arrest increasing risk, Int. J. Disast. Risk Reduc., 31, 1013–1029, https://doi.org/10.1016/j.ijdrr.2018.09.001, 2018.
Susi, T., Johannesson, M., and Backlund, P.: Serious Games – An Overview, Elearning, 73, 1–28, 2007.
Terti, G., Ruin, I., Anquetin, S., and Gourley, J. J.: Dynamic vulnerability factors for impact-based flash flood prediction, Nat. Hazards, 79, 1481–1497, https://doi.org/10.1007/s11069-015-1910-8, 2015.
Terti, G., Ruin, I., Anquetin, S., and Gourley, J. J.: A Situation-Based Analysis of Flash Flood Fatalities in the United States, B. Am. Meteorol. Soc., 98, 333–345, https://doi.org/10.1175/BAMS-D-15-00276.1, 2017.
Terti, G., Ruin, I., Gourley, J. J., Kirstetter, P., Flaming, Z., Blanchet, J., Arthur, Z., and Anquetin, S.: Towards Probabilistic Prediction of Flash Flood Human Impacts, Risk Anal., 39, 140–161, https://doi.org/10.1111/risa.12921, 2019.
Thielen, J., Bartholmes, J., Ramos, M.-H., and de Roo, A.: The European Flood Alert System – Part 1: Concept and development, Hydrol. Earth Syst. Sci., 13, 125–140, https://doi.org/10.5194/hess-13-125-2009, 2009.
Turkay, S. and Adinolf, S.: What do players (think they) learn in games?, Procedia – Soc. Behav. Sci., 46, 3345–3349, https://doi.org/10.1016/j.sbspro.2012.06.064, 2012.
Tychsen, A., Hitchens, M., Brolund, T., and Kavakli, M.: THE GAME MASTER, in: Proceedings of the Second Australasian Conference on Interactive Entertainment, Sydney, Australia, 215–222, 2007.
UK Met Office, Deltares, & Red Cross Red Crescent Climate Centre, available at: https://understandrisk.org/wp-content/uploads/FBEA-Game-v5-1.pdf (last access: 11 March 2019), 2018.
Van Ruijven, T. W. J.: Serious games as experiments for emergency management research: a review, in: ISCRAM 2011: Proceedings of the 8th International Conference on Information Systems for Crisis Response and Management, 8–11 May 2011, Lisbon, Portugal, 2011.
Walker, W. E., Giddings, J., and Armstrong, S.: Environment, Training and learning for crisis management using a virtual simulation/gaming, Cogn. Technol. Work, 13, 163–173, https://doi.org/10.1007/s10111-011-0176-5, 2011.
WMO: WMO Guidelines on Multi-Hazard Impact-based Forecast and Warning Services, available at: https://www.wmo.int/pages/prog/www/DPFS/Meetings/ET-OWFPS_Montreal2016 (last access: 11 March 2019), 2015.