This paper presents the findings from a study aimed at understanding whether video games (or serious games) can be effective in enhancing volcanic hazard education and communication. Using the eastern Caribbean island of St. Vincent, we have developed a video game – St. Vincent's Volcano – for use in existing volcano education and outreach sessions. Its twin aims are to improve residents' knowledge of potential future eruptive hazards (ash fall, pyroclastic flows and lahars) and to integrate traditional methods of education in a more interactive manner. Here, we discuss the process of game development including concept design through to the final implementation on St. Vincent. Preliminary results obtained from the final implementation (through pre- and post-test knowledge quizzes) for both student and adult participants provide indications that a video game of this style may be effective in improving a learner's knowledge. Both groups of participants demonstrated a post-test increase in their knowledge quiz score of 9.3 % for adults and 8.3 % for students and, when plotted as learning gains (Hake, 1998), show similar overall improvements (0.11 for adults and 0.09 for students). These preliminary findings may provide a sound foundation for the increased integration of emerging technologies within traditional education sessions. This paper also shares some of the challenges and lessons learnt throughout the development and testing processes and provides recommendations for researchers looking to pursue a similar study.
Education and communication play a vital role in improving the resilience of vulnerable populations at risk from natural disasters (Johnston et al., 1999; Ronan and Johnston, 2003; Shaw et al., 2004; Paton et al., 2008). Conventionally, such awareness-raising activities are delivered in a number of guises – typically leaflets, posters, presentations, maps, TV and radio broadcasts. Often these educational products are aimed at school-age children, in part due to ease of access and in part reflecting current thinking that children filter information through to their parents through informal conversations (Ronan and Johnston, 2003; Carlino et al., 2008; Sharpe and Izadkhah, 2014). However, it is becoming increasingly important to evolve existing education and communication techniques to better engage a new generation of learners. It has been argued that, with advancements in technology, individuals today learn in ways different to their predecessors (Prensky, 2001; Annetta, 2008; Bekebrede et al., 2011; Sharp, 2012). A fresh generation of learners – sometimes called the Net Generation or Net Gen (Tapscott, 1998) – are accustomed to a digital age in which information, news and entertainment are obtained instantaneously and delivered directly to them on personal devices (e.g. mobile phones, tablets and laptops). This has led to a rise in innovative teaching techniques in the classroom, such as the use of video games, in an effort to better motivate this new generation to learn (Prenksy, 2001).
This paper will focus on how educational video games can be used as a tool for public outreach around raising awareness of volcano hazards with at-risk communities. It reflects a recent surge in the application of so-called “serious games” – video games with the primary purpose of education, not entertainment – for the purposes of learning and training (Michael and Chen, 2005; Zyda, 2005; Djaouti et al., 2011), ranging from applications in medicine to military training and spanning everything from personal health to curriculum education. The paper considers the emergence of serious games in the realm of natural hazard education and critically examines their role for communicating volcanic hazards. Highlighting a lack of empirical evidence to demonstrate that geohazard-related serious games promote more effective learning, the paper presents the development and testing of a serious game specifically designed to test volcanic hazard awareness among schoolchildren and adults on the Caribbean island of St. Vincent. The study provides a preliminary assessment of the effectiveness of virtual environments as a learning technique and discusses the practical issues and challenges encountered when conducting this type of research.
In 2015, the United Nations formalised The Sendai Framework for Disaster Risk Reduction 2015–2030, with the goal of reducing the number of deaths, injuries and impacts caused globally by disasters (human and natural). To address that goal, The Sendai Framework identifies the need for participating countries to “strengthen public education and awareness in disaster risk reduction”, specifically promoting the use of social media and community mobilisation campaigns and encouraging the education of all at-risk communities (UNISDR, 2015). This reflects an acceptance within the disaster risk reduction arena that education and outreach programmes can prepare hazard-prone communities (McKay, 1984; Ronan and Johnston, 2001; Paton et al., 2008; Johnston et al., 1999; Paton et al., 2000),
Although it is widely accepted within the disaster risk reduction arena that education and outreach programmes can be successful in reducing community vulnerability (McKay, 1984; Johnston et al., 1999; Paton et al., 2000; Ronan and Johnston, 2001; Paton et al., 2008), it is also argued that a better educated population does not necessarily lead to the adoption of preparation measures (e.g. stockpiling bottled water and food or developing a family emergency plan) (Sims and Baumann, 1983; Paton et al., 2000; Paton, 2003; Kirschenbaum, 2005; Haynes et al., 2008). The process for at-risk communities to adopt preparative measures is multifaceted and many studies have proved that in some cases communities actually reduce their preparation measures after educational intervention (Johnston et al., 1999; Shaw et al., 2004; Paton et al., 2008). Research has shown that as the probability of a major hazardous event occurring, such as a volcanic eruption, posing a threat to an at-risk community is relatively low, this can lead to the development of a “not in my lifetime!” attitude of residents. Often the threat from an imposing hazard is low, leading to communities becoming unmotivated to adopt preparation measures and further beginning to rely on scientists to give enough warning before an event occurs to prepare themselves (Johnston et al., 1999; Shaw et al., 2004; Paton et al., 2008).
The disassociation between educating communities and their adoption of preparedness measures has now led to a corresponding shift towards making these education programmes more effective. The emergence of the leading role of social media highlights the educational advancements of digital technology that could provide the “stepping stone” between education and the adoption of preparedness measures. Examples of this are the use of sophisticated GIS programmes to replace 2-D maps with 3-D terrain models and the popularity of novel creative media for virtual reality animations.
Maps are one of the most common tools for education in hazardous areas, whether they are topographical maps to demonstrate landscape features or hazard maps to show areas most at risk from being affected by a natural hazard. Nevertheless, the use of maps to communicate hazard to at-risk communities can encounter problems, notably with high levels of illiteracy, comprehension, language and terminology (Donovan 2010). Haynes et al. (2007) explored local residents' comprehension of existing 2-D maps and newly developed 3-D maps for volcanic hazard mapping on the Caribbean island of Montserrat. Two resident groups were used, with the first asked to locate themselves and landmarks on 2-D maps and the second using 3-D topographic maps. Both groups were also supplied with oblique aerial photographs. The study showed there was a minor improvement for the participants using the 3-D maps in their ability to locate themselves. However, the most significant improvement was in the understanding of the relationship between hazard and topography using the aerial photographs. In a similar study for lahar hazards at Mount Hood, Oregon, USA, participants were shown four maps and asked to complete activities for terrain interpretation, estimation of lahar travel times and evacuation routes (Preppernau and Jenny, 2015). The study showed that most participants preferred the use of 3-D maps, being able to interpret the terrain better and choose the more appropriate evacuation routes.
A logical progression of this realisation is how we may be able to improve hazard awareness using better-resolved 3-D representations of hazardous settings as well as understanding whether we can use virtual reality environments to foster even more effective spatial thinking. Computer games are increasingly offering that potential, hence the emergence of serious games, defined by Michael and Chen (2005) as a game in which “education is the primary goal, not entertainment, with the intention of improving a specific aspect of learning.” To be effective as a learning tool, a serious game needs to be carefully designed and underpinned with a sound and compatible learning theory. The rise of serious games has been developed as a potential way to engage a new generation of classroom learners – N-Gen or Net-Gen as first coined by Tapscott (1998) – who are characterised by a more technological way of life (Prensky, 2001).
The potential for serious games in disaster reduction and outreach is highlighted by the fact that one of the first purpose-built computer games was created by the UN International Strategy for Disaster Reduction (ISDR) as part of the Hyogo Framework for Action. The ISDR game, Stop Disasters!, was designed to educate children about preparing for disasters by building resilient communities for a number of hazards (e.g. hurricanes, earthquakes, wildfires and flooding). The game was supported by a website of teaching materials and has been used in education programmes by the Seismic Research Centre (SRC) in the Caribbean region (R. Robertson, SRC, Personal Communication, 2015). A similar approach was taken by UNESCO and the government of Japan in designing Sai Fah: The Flood Fighter, a bespoke game built as a response to devastating floods in Thailand in 2011. Like Stop Disasters!, the game is a simple platform aimed at children, intended to educate them about how to recognise when flooding is likely and about the actions they should take to prepare. The game is freely available for a number of devices and has been translated into several languages for use in education programmes around the world.
Bespoke designed video games such as Sai Fah: The Flood Fighter and Stop Disasters! allow for a more interactive, engaging and tailored educational experience for the players. Another advantage is the ability of games to provide instantaneous feedback to the player, which means that misconceptions and misappropriation of knowledge can be avoided at an early stage. As is commonplace in outreach activities, the target audience for both games is children, but the widespread and popular use of video games means that adult learners can benefit too. Sai Fah is available on many mobile devices and is run primarily as a stand-alone application. This means that adults have the access and means to play the game to make themselves more aware.
Sai Fah: The Flood Fighter and Stop Disasters! possess common characteristics that make them engaging for the player, including high levels of interactivity (through clickable icons and activities), a clear game objective and reward-based activities (e.g. point scoring and upgrades). The use of video games such as these and other more creative education techniques (e.g. comic books, puppet shows and theatrical role play) are not often used within normal day-to-day classroom activities. Therefore, their use in outreach activities provides a novelty factor which can motivate and engage participants to learn. The common characteristics identified provide some considerations for the future development of creative approaches to education and outreach practices.
In summary, given that video games are emerging as a popular tool in disaster reduction education, it would seem timely to critically appraise whether such serious gaming offers an effective means of strengthening public awareness more broadly among hazard-prone communities. This paper does this in the context of communities on the eastern Caribbean island of St Vincent, where volcanic activity threatens a population largely unfamiliar with its most extreme eruptive potential.
St. Vincent is the largest island in an archipelago that forms the eastern Caribbean nation of St. Vincent and the Grenadines. Most of its 110 000 inhabitants (World Bank, 2016) live in the capital Kingstown, located to the south of the island. The northern part is occupied by an active volcanic centre, La Soufrière, which has a violent eruptive history, with significant explosive eruptions in 1718, 1812, 1902 and 1979. Contemporary accounts and photographs of the 1902 eruption by Anderson and Flett (1903) show that much of the north of the island, including its extensive plantations, was devastated and over 1500 people were killed. The 1979 eruption of La Soufrière was of a significantly smaller scale, causing no fatalities but forcing the evacuation of 20 000 people to shelters in the south, where they remained for up to 4 months after the eruption. Since 1979, the volcano has entered a state of quiescence and shows very little signs of life with only minor fumarolic activity within the summit crater.
Although many islanders are aware of the 1979 eruption, over half (56 %) (World Bank, 2016) of the population are under the age of 35, meaning they have no direct experience of volcanic eruptions on the island. In contrast, with St. Vincent prone to many other natural hazards – experiencing hurricanes, flooding, earthquakes and landsliding on a more frequent basis – the island's residents tend to prioritise preparedness at the expense of volcanic threats, which are deemed to pose less of a risk to day-to-day life. This is a phenomena also previously noted in other areas prone to multiple natural hazards (e.g. Shaw et al., 2004; Perry and Lindell, 2008). On St. Vincent, therefore, effectively raising awareness of volcanic hazards is vital to ensure the population is motivated to prepare for a potential future eruption of La Soufrière, which could occur with little warning.
In the event of a volcanic eruption of La Soufrière it is the responsibility of the National Emergency Management Organisation (NEMO) to coordinate the emergency response. NEMO is a government department with the role of disaster preparation, mitigation and management for St. Vincent and the Grenadines. NEMO have developed the National Emergency Plan for a variety of natural hazards that threaten the island and undertake a continual programme of public education to encourage community preparedness. In addition, the Seismic Research Centre (SRC) at the University of the West Indies monitors La Soufrière, in partnership with the Soufrière Monitoring Unit (SMU), via a remote network of instruments around the island and has responsibility for providing information regarding the volcano to the government of St. Vincent during a potential or developing volcanic crisis.
Together, NEMO and SRC coordinate an annual programme of education and
outreach designed to commemorate the anniversary of the 1979 eruption (13 April). Called Volcano Awareness Week (VAW) this week-long programme of
activities aims to supplement elements of volcano education included within
different levels of the national school curriculum. The principal activities
include the following:
visits to many primary and secondary schools across the island to give education sessions; volcano hikes accompanied by local geologists and people that monitor the volcano; community-based outreach sessions open to the public; review of the National Disaster Emergency Management Plan by stakeholders.
Summary of the key ideas and repetitive themes from the user requirements data collection focus groups and online questionnaires.
The public outreach sessions present general information about the formation of volcanoes in the Caribbean as well as specific information about hazards that may occur during a future eruption of La Soufrière. Robertson (2005) developed a volcanic hazard map for St. Vincent, highlighting the areas that are likely to be affected using traffic light colour coding (where red is the most hazardous and green the least hazardous). This hazard map is widely circulated across the island and a reference copy is printed within the island's telephone directory. The hazard map is also included in a SRC leaflet “Volcanoes of St. Vincent”, which explains the volcano, its history and the monitoring network. This leaflet is often distributed during outreach sessions and campaigns across St. Vincent. NEMO and SRC also often conduct television and radio broadcasts during outreach campaigns to publicise their work and promote awareness.
St. Vincent's active and diverse programme of outreach activities provides an appropriate backdrop to appraise the efficacy of emerging computer-based depictions of volcanic hazards against conventional volcano hazard education efforts. The following section describes how a bespoke computer game, St. Vincent's Volcano, was designed based on information collected from ground surveys, storyboarded on the basis of La Soufrière's historical eruptions and developed over the course of a year by a team of developers at Plymouth University to produce an interactive game. Later sections report on the experience of trialling St. Vincent's Volcano with small user groups on the island to road test the software and identify technical improvements for future iterations of the game.
For this study a serious game called “St. Vincent's Volcano” was developed. In this section we describe the process undertaken during the game's development from design to completion which comprised three stages: establishing user requirements, storyboarding the design and game development.
The initial phase of the game design process was to establish the end user requirements. During August 2014, two focus groups were held in two communities in close proximity to the La Soufrière volcano (Owia and Petit Bordel). The aim of these focus groups was to establish what community members and future potential end users would like to see included within the game and to establish some basic technological requirements (e.g. desired platform and facilities available). In addition, an online questionnaire of requirements was emailed to key stakeholders in volcano education on St. Vincent (UWI SRC, NEMO, SMU and The Red Cross). A copy of the questionnaire is included as Appendix A. The main recurring themes from the focus group discussions and the completed questionnaires are summarised in Table 1.
This information from the St. Vincent community focus groups and stakeholder questionnaires established the basic concept of the game. The community groups expressed their desire for the inclusion of historical eruptions of La Soufrière, with particular reference to the last eruption in 1979. From the stakeholder questionnaires, the inclusion of information relating to the three primary volcanic hazards that are likely to occur during future activity (pyroclastic flows, lahars and ash fall) was considered to be important. Both groups of participants agreed that the primary target audience was secondary-school-age children. Further justification provided by the stakeholder questionnaires identified that all secondary schoolchildren on St. Vincent are provided with a personal laptop by the government as part of their studies.
The second phase in the design of St. Vincent's Volcano was storyboarding the game concept as a communication pathway between designers and developers. A series of storyboards were created for each game scene, providing detailed descriptions on the look-and-feel nature of player interactivity and navigation through the game. They also included all textural and oral descriptions and detailed information snippets of how the hazardous phenomena (ash fall, pyroclastic flows and lahars) behave to enable realistic-looking visualisations to be developed.
Based on the gathered information from the online questionnaires and focus groups, visualisation of the two most recent historical eruptions, based on the 1902 and 1979 eruptions, were integrated into the game. The reconstructive visualisations were designed to illustrate the timeline of actual events during each eruption and are accompanied by textual descriptions and oral descriptions via voice-over “radio” recordings. The three volcanic hazards identified by the stakeholders were integrated into interactive eruptive scenes with visualisations of each phenomena augmented by clickable icons that reveal brief dialogue boxes that explain, in simple terms, the formation and behaviour of that hazard.
A serious game must have a robust pedagogic underpinning. In the case of the St. Vincent's Volcano game, the primary pedagogical method used is Kolb's model for experiential learning; this model involves a learner transforming an experience into concrete knowledge, which they are then able to apply to future learning experiences (Kolb, 1984; Vince, 1998; Bellotti et al., 2013; Konak et al., 2014). Four stages make up the learning cycle: concrete experience, reflective observation, abstract conceptualisation and active experimentation. In the context of St. Vincent's Volcano, experiential learning is achieved though players being introduced to key themes (volcanic hazards) early on in the game by watching historical eruption visualisations (concrete experience). The player then interacts with scenes where the same visualisations of the volcanic hazards are used but this time with added information and detailed descriptions of their formation and behaviour (reflective observation and abstract conceptualisation). The final stage is the players engaging with a multiple-choice quiz in which they can apply their newly obtained knowledge. For each correct question the player is awarded points and where questions are answered incorrectly, instantaneous feedback appears to explain why the answer is incorrect (active experiment). The game concludes with the player being told their overall scores and awarded a medal (gold, silver or bronze), depending on their score.
A secondary learning theory embedded in the game development is Sweller's cognitive load theory, which is based on the hypothesis that for effective learning, a person's short-term memory can process simultaneously only a certain number of elements (Chandler and Sweller, 1991; Sweller, 1994; Bellotti et al., 2013). Ensuring that the participant's attention was not unduly overloaded was achieved in the game by eliminating repetitive information to reduce redundancy and engaging both auditory and visual senses to increase the working memory capacity (Huang and Tettegah, 2010).
The game development was an iterative process fed by continual feedback and testing between designers and developers throughout the 1-year development process. The finalised version of the game varied little from the storyboarded design, with minor differences due to technical constraints of the software used (Unity 3-D).
The completed St. Vincent's Volcano game (Fig 1a-1d.) incorporates user requirements, learning theory and established volcanic hazard communication information (e.g. volcanic hazard map). Three geographic perspectives are adopted, reflecting the three most populated centres in proximity to the volcano: Chateaubelair, Georgetown and Fancy. In each of these perspectives, there are four main scenes.
Interactive island model: the player can manipulate
a top-down-view model of the island to identify where they live in proximity
to the volcano. The model is a realistic visualisation built using digital
elevation data and satellite imagery. The players can add the volcanic hazard
map with detailed definitions to the model to determine which hazard zone
they live in. This scene is designed to be interactive with the player able
to manipulate the model to explore the island. Historical eruption visualisations: La Soufrière is
brought to life through a series of visualisations depicting two historic
eruptions (1902 and 1979). The visualisations are based on the historical
record and people's personal accounts of the eruptions. The visualisations
are accompanied by textural descriptions of the activity and oral
descriptions provided as “radio broadcasts”. Hazard training: three scenes of visualisations
depicting the potential future hazards phenomena (ash fall, pyroclastic flows
and lahars). The player is guided through via five clickable icons in each
scene, which reveal snippets of information about the formation and behaviour
of each phenomenon. Multiple-choice game quiz: all questions relate to
information given throughout the game and instantaneous feedback is provided
where a question is answered incorrectly. All answers are recorded through
in-built game analytics (if players opts in). This quiz is not designed to
measure learning, rather to identify topics of strength and weakness and give
details on how players progress though the game.
Screenshots from the St. Vincent's Volcano game.
To ensure that the volcano game was appropriate for the target audience, St. Vincent's Volcano was trialled on the island over a 6-week period (March to May 2015), which coincided with Volcano Awareness Week (VAW). Two types of outreach session were organised – student learners and adult learners.
The Ministry of Education for St. Vincent and NEMO recruited schools through a circular email prior to the VAW activities, explaining the study using the game. For the 2015 VAW activities, schools primarily in the green volcanic hazard zone (based on the established volcanic hazards of Robertson, 2005) were targeted for outreach – primarily schools located in the Kingstown area of the island and surrounding towns. Since many students travel to the capital, Kingstown, for school but often do not reside there, all students participating in the study were asked to provide their location of residency on St. Vincent (Fig. 2). Due to this factor, influence from targeting students from one particular hazard zone – in this instance the Green Zone – is not deemed to have had an effect on the data as many participants reside in different hazard zones (red, orange or yellow).
Map showing the location of residency for most participants used in
this study. The number of participants stated is the number of completed and
usable data sets of which provided location of residency information
(students, who participated in all types of sessions,
During the study, 126 students from 13 secondary schools participated, providing 65 usable data sets of participants who played the game (once the data had been cleaned to remove incomplete data and cases where cheating was evident). Sessions typically comprised 20–35 grade 4 students (14–15 years old), which was deemed optimal for the study. Each session was run dependent on the time slot and facilities available and, the number of students per session. The students were either asked to listen to a presentation by a member of the UWI SRC outreach team on volcanic hazards and St. Vincent, play the St. Vincent's Volcano game or do both. Students were able to ask questions throughout the session from any member of the outreach team and support was given where required.
One session comprising eight students was run as a control group, where the students received only a traditional outreach session (presentation only). The presentation some of the students received included basic information on the tectonic regime in the Caribbean and its link to the formation of volcanoes in the region, the historical eruptions of La Soufrière and descriptions of the types of eruptive phenomena that could be experienced on St. Vincent in the future (e.g. pyroclastic flows, lahars and ash fall). Further information included how the volcano is monitored and which organisations are responsible for this (UWI SRC and SMU).
As part of the grade 4 education, students cover some of the basics of volcanoes as part of the normal curricular activities. The syllabus covers the formation of volcanoes and some of the products generated during a volcanic eruption. All student participants, at the time of the study, had completed their curricular studies on volcanoes within their normal geography classes.
With the assistance of community leaders, four adult sessions were organised
across the island in the towns of Georgetown, Chateaubelair, Fancy and
Kingstown with a total of 25 participants (23 usable data sets once the data
had been cleaned to remove incomplete data and cases where cheating was
evident). The participants were recruited through word of mouth and
direct invitation by community leaders. Adult sessions were held in a variety
of locations including a preschool, secondary school, community centre and
civil offices. The participants ranged in age (between 18 and 60
During the adult sessions, the participants were asked only to play the game with no supporting presentation. Participants were able to ask questions throughout the session and assistance was given to those that found difficulty with using the game (associated with low levels of computer literacy). After each session, the participants were fully debriefed and able to ask any further questions they had relating to the information they had received.
The implementation sessions provided a valuable opportunity to collect data on the responses of those playing the volcano game, which could be used to evaluate its effectiveness as a learning or awareness-raising tool. Qualitative and quantitative data on player responses were obtained through three contrasting methods: pre- and post-test knowledge quizzes, in-built game analytics and video recordings.
The main source of data collection was through the completion of knowledge quizzes before and after all of the sessions took place. The knowledge quizzes comprised 12 open-ended questions relating to general terminology and definitions of volcanic phenomena and to the historical eruptive history of St. Vincent. The pre- and post-test knowledge quizzes comprised the same questions, although they were slightly reworded and the order of questions was altered to deter participants from memorising an idealised answer. Copies of the knowledge quiz for both adults and students are included in Appendix B. Depending on the participant's preference, the knowledge quizzes were completed either manually or digitally, were undertaken independently and took approximately 20 min to complete. All post-test knowledge quizzes were completed immediately after completion of the intervention sessions.
Questions within the knowledge quizzes covered one of three topics: firstly, general knowledge about the volcanic history of St. Vincent (e.g. the name of the volcano or previous eruption dates) and, secondly, detailed information on volcanic hazards (e.g. what is meant by a pyroclastic flow or when a lahar can occur) to establish depth of knowledge and participants' use of colloquial language. Finally, questions relating to the existing volcanic hazard map (e.g. what hazard zone is Fancy in) to assess the students' exposure to educational resources and to test whether the game enhances how this information is received.
Quantitative data were complimented by qualitative data collected during session observations and video recordings. These observations and video recordings were primarily undertaken to enable analysis on levels of engagement and motivation throughout the sessions and confirm specific observations (e.g. behavioural responses). They also facilitate data collection, e.g. from discussions with other participants, talking-out-loud phases and other interactions between participants which could be missed when running a busy session.
For the final multiple-choice question section of St. Vincent's Volcano game, the participants could opt-in to having their progress recorded automatically through in-built game analytics. Gaming analytics can help identify the subject areas where participants are weakest, therefore allowing for a more tailored education session, as well as providing information about how the game is played and how it can be improved further. When opted in, information on how long each participant spent playing the game, the questions they were asked in the multiple-choice quiz and their responses was recorded. Unfortunately, for many of the student participants the analytics malfunctioned (due to human error) or it was not possible to retrieve the data; however, most of the adult participants' analytics were successfully recorded.
To have confidence in the evaluation of St Vincent's Volcano as a learning tool, the quality of the participant data is paramount. One of the school sessions comprising eight students was used as a control experiment; these students only received the outreach presentation and did not play the game, but the same data were collected from this session. In total, 65 fully completed data sets of the participants who played the game could be used in this study, representing 51 % of the original participant pool. The gender ratio for the student participants has not been recorded as it is not deemed to have an influence on the results, since all schoolchildren receive the same education on St. Vincent. For the adult sessions, two data sets were removed (due to one participant not completing the post-test knowledge quiz and one being observed using the internet to complete the knowledge quiz), meaning that 23 data sets were used for this study.
This section presents preliminary results of implementation testing on St. Vincent. As this research is ongoing, not all aspects of the data collection have been analysed. Instead, this section presents the preliminary results of the knowledge quizzes, session observations and preliminary results from the in-built game analytics for the adult participants.
Both student and adult participants completed a pre-test and post-test knowledge quiz during their session. The sessions took three primary forms: either a volcanic hazards presentation, playing of the St. Vincent's Volcano game or a combined session of the presentation and the game. The percentage completion for the two knowledge quizzes demonstrates an increase in percentage of questions answered between the pre-test and post-test quizzes of 12.8 % for student participants and 17.7 % for adult participants.
All knowledge quizzes were allocated a unique reference number to remove any
chance of bias during marking. A rubric was developed for marking which
allowed points to be awarded for correct answers and for additional points to
be awarded where deeper knowledge was demonstrated, such as where a student
can add specific details relating to a hazardous phenomenon (e.g. the speed at
which a pyroclastic flow can travel or the size of ash particles in
millimetres). The two scores for the pre-test and post-test knowledge quizzes
were then plotted against each other as percentages for both the student data
sets (
Graphs showing the preliminary pre-test and post-test knowledge quiz
score marks in percentage for
Both graphs show a general positive trend for this preliminary data, demonstrating that the participants' knowledge improved after they had received an education session. The dashed line on both graphs is the “line of no improvement”, i.e. any data points above this line indicate that participants improved their knowledge between the pre- and post-test knowledge quizzes, and any on or below this line did not improve their knowledge. For the adult data sets there are two participants that fall on or below the line of improvement compared with five student learners. When calculated, the average increase in score for the adult participants is 9.3 % compared to 8.3 % for the student participants.
The
To assess an individual's learning gain, we adopt Hake's (1998) normalised technique of calculating learner's knowledge change. Figure 4 shows the pre-test scores (%) for both adult and student participants plotted against their learning gain – expressed as the difference between pre-test and post-test score (%), divided by the difference between the maximum possible score (100 %) and the pre-test score. This method determines potential gains each participant can make irrespective of their initial starting level.
The average learning gains have also been calculated for the student data set
and for the adult data set. The average learning gains for the adult
participants (0.11
Graph showing the preliminary learning gains for both adult and student participants and the averages for both populations. Overall, both sets of participants show positive learning gains with the average adult learning gains (0.11) similar to those of the student participants (0.09).
Throughout both the adult and student sessions, observations were made about how the participants were engaging with the volcano game. For the student participants it was often difficult to make specific observations due to the number of students per session; therefore, these observation data relate to the general attitudes and behaviour of the class as a whole, rather than to individuals.
When students played the St. Vincent's Volcano game, in general they were noted to be engaged and displayed an active interest in what they were seeing (asking and answering questions throughout). Some participants asked questions about the information displayed within the game and many were noted to be discussing the game with a neighbour. Some students were able to move more quickly through the game than others and, as a result, as the session progressed, less discussion was noted but all students remained on task. During the conclusion of the sessions, students were asked if they had any questions and in most sessions no questions were asked. In one session, a participant asked about the future of the game and how it could be accessed again and when asked generally if they enjoyed the game, the response from all participants was positive.
In general, the adult participants displayed good motivation and high engagement levels with the game, although many participants required assistance with navigation and interactivity throughout the session. During the adult sessions, three notable methods of engagement were observed: (1) two participants were observed writing notes as they moved through the game, (2) many of the participants asked questions throughout and (3) four participants linked what they saw in the game to their personal experiences.
At the end of the sessions, many of the participants asked questions about things they had observed in the game or had been asked as part of the knowledge quizzes. In the adult session held in Fancy, many of the participants had experienced the 1979 eruption. One participant during this sessions was noted to have an emotive memory recall – suggesting that the game brought back strong memories about the events they had witnessed in 1979. In this case the emotive trigger was noted to be the visualisation of pyroclastic flows down the volcano flanks. Further, at the conclusion of playing the game, many of the participants in this session who had experienced the 1979 eruption began to retell their stories of the events.
As detailed in Sect. 4.3, in-built game analytics were recorded for adult
participants only (
Graph showing the preliminary results for the data retrieved from
in-built gaming analytics for the adult participants (
Adults analytics data recorded from the sessions included the length of time
each participant took to complete the game. These data were plotted in
minutes, against the score difference between the pre- and post-test knowledge
quizzes (%), then characterised by age category (Fig. 5). Figure 5 shows
that the majority of participants in the age categories < 39 years
(
Preliminary results for the field testing of the St. Vincent's Volcano game demonstrate a general improvement in knowledge quiz scores for both adult and student participants. The improvement measured was similar for both adult and student participants, with a post-test increase in correct responses of 9.3 and 8.3 % respectively. The same tendency is observed when the pre-test score percentage is plotted against learning gains (Fig. 4), with average learning gains of 0.11 for adults and 0.09 for students. This is a commonly observed trend for participants who have undertaken any form of intervention (Ronan and Johnston, 2003; Shaw et al., 2004; Johnston et al., 1999; Paton et al., 2008). To establish the effectiveness of the game as an education and communication tool, it must be compared to an existing technique.
A comparison between a game session and a traditional session is only
possible for the student participants. One control session for student
participants was undertaken comprising only a presentation (traditional
method), the same presentation as used with other student participants for
consistency. From this session, eight student data sets were recorded, which
demonstrate average learning gains of 0.12
Observations taken throughout the sessions identified some indications of good motivation and engagement. The students were noted to be discussing the game with neighbours, asking and answering questions and staying on task throughout the sessions. A more detailed analysis of the video recordings is required to further constrain the levels of motivation and engagement achieved with more traditional methods and the role it has on the longevity of their knowledge gain.
Levels of motivation and engagement were expressed slightly differently for the adult participants. Some participants were observed writing notes, asking questions throughout the sessions or linking what they saw within the game to their personal experiences of 1979. In one case a strong emotive memory recall was noted as the participant played the game, suggesting that the game's levels of realism and the fidelity of the visualisations are significant enough to trigger this reaction. In other cases, on completion of the game, participants began to recall their stories of 1979. This strong emotive memory recall may be significant for future iterations of the game or for researchers considering a similar venture as this can lead to memory encoding, thus a deeper learning experience (LeBar and Cabeza, 2006; de Byl, 2015). Further research is required using the video recordings to constrain the triggers of the memory recall and to understand the role of both realism and familiarity on the educational experience.
During the implementation, two (3.0 %) student participants and one (4.3 %) adult participant recorded negative gains between the two knowledge quizzes. Although the exact cause of this is uncertain, some participants were observed losing engagement with the session due to the repetitive nature of completing knowledge quizzes and the duration of the session. The video recordings taken during the sessions will be examined to determine if there is any further variable that could be controlling this result.
Of interest, with respect to the data collection, is that over a quarter (27.7 %) of participants' data were removed due to cheating or copying from a neighbour (particularly amongst the student participants). Many of these had used the internet to source answers when completing a digital version of the knowledge tests. This could add weight to anecdotal arguments that a new generation of learners exist in today's classrooms where, when an answer is unknown, the knowledge is instantly acquired from digital sources. Although cheating is an inevitable problem in this kind of study, it has reiterated the need for manual completion of all knowledge tests in future iterations of the study. Further, a more explicit explanation of the need for independent completion of the knowledge quizzes may also reduce the number of data sets being removed.
A general trend of improved knowledge after intervention is observed for
96.5 % (
Throughout the different phases of this study a number of challenges have arisen. Here, we provide some insight into some of these issues and offer some recommendations for researchers considering pursuing a similar line of research.
During the development process some technical issues arose. They ranged from the ability to correctly and realistically visualise the hazardous phenomena to the interactivity and flow of the game. These issues were overcome through an iterative approach to the game development with multiple phases of trials and tests.
During the study, 13 secondary schools were visited on St. Vincent to take part in the study however, data could only be collected from six due to issues relating with the testing strategy for the game. The testing strategy was dependent on a number of factors: the number of students per session (ratio of instructors to students), the time available to run the session, the basic facilities available in schools and the number of laptops available per session. For the two schools that could not be sampled, one had class sizes of over 100 students, which was too large for the study to be run, and a second did not have electricity within the classroom. Many of these factors were not accounted for during the testing strategy design, which led to the running of three dynamic styles of session (presentation only, game only and presentation and game) as a way to overcome these issues without compromising students' education or the data collection. For future studies where similar issues are encountered, a flexible testing strategy that allows for data to be collected from large groups or where facilities are limited should be developed to allow for more effective data collection. For example, this could comprise video recordings for large groups to monitor interactions and engagement or simple surveys which ask participants to reflect on their own learning from the session.
As previously stated in Sect. 5.1, some of the completed data sets had to be removed from the study due to evidence of cheating, either copying from a neighbour or using the internet. Due to the size of some of the sessions, copying and discussing answers was often difficult to control despite instructions explaining the need for independent work. When participants did not know the answer to a question they were instructed to leave it blank or write that they did not know. Despite this, a surprising number of the participants' answers to quiz questions were noted to be copied, in most cases word for word from the internet. For researchers considering a similar line of research, it would be prudent to ensure all participants manually complete knowledge quizzes under exam conditions (where possible) and emphasise the importance of the research of which the participants are undertaking and their role within the study.
General feedback was compiled for the game from both stakeholders and participants during field testing on St. Vincent. Many stakeholders (teachers, UWI SRC, NEMO and the Ministry of Education) expressed their desire to integrate the St. Vincent's Volcano game into their existing education sessions in the future. Participants also expressed their overall satisfaction with the game but with many suggesting improvements that could make it more engaging for them (e.g. higher levels of interactivity and simplified navigation). Some technical issues were also identified during field testing (e.g. sound issues, mouse navigation problems and analytics malfunctions).
The research team, depending on funding, aim to improve the game through incorporating participant feedback and fixing technical glitches identified during field testing. Once the modifications are made, the intention is to publish the completed game online as a freely available resource, ideally hosted on a website with supporting information (e.g. preparation and mitigation measures) to ensure the game's longevity. The final online game will require minimal maintenance as it primarily includes generic information on hazardous phenomena which does not alter over time. By being freely available online, it is hoped the game can be used as a tool for education and outreach around the world.
The preliminary findings from this study suggest that serious games have the potential to be effective tools in volcano education for both traditional stakeholder groups (school students) and also for non-traditional stakeholders (adults). Serious games, therefore, would seem to be a promising communication and educational technique, but this novel approach faces a number of challenges. Arguably, the main challenge is that game development is an expensive and time-consuming process. The St. Vincent's Volcano game was the work of an in-house team of developers at Plymouth University over a 1-year period, during which it evolved via multiple iterations as the limitations of the software (and the designers) were overcome. Technical challenges ranged from the ability to correctly and realistically visualise the hazardous phenomena to the interactivity and flow of the game. Although the completed volcano game was not intended to be of an equivalent quality to commercial computer games, the finished product still had to be fully functional, robust and interesting to play. That requires a development process that is longer, more complicated and more costly than conventional volcano awareness-raising measures.
Nevertheless, the early indications from this study are that volcano-based computer games can increase uptake of hazard communication information. However, while the more traditional methods of education, such as maps and leaflets, typically target particular groups (especially school-age children) (Ronan and Johnston, 2001), the widespread popularity of computer gaming offers the opportunity to extend the reach to older and harder-to-reach demographic groups. This relates to the growing recognition that with advancements in technology, people today are taking in information in different ways to their predecessors. Perhaps more significantly, it is often difficult to establish the effectiveness of outreach sessions or provide any follow-up to consolidate learning due to time and funding constraints of outreach organisations, but bespoke computer games – and the analytics within – open up the prospect of evaluating the effectiveness of hazard awareness and disaster preparedness.
The St. Vincent's Volcano serious game is an attempt to bridge the gap between these different intergenerational learning styles and also to overcome some of the problems commonly encountered in conventional volcanic hazard education (e.g. language barrier, varying educational backgrounds and the use of complex and difficult-to-understand maps and diagrams). Its twin aims are to improve knowledge of future potential volcanic hazards on St. Vincent and to integrate traditional methods of education in a more interactive manner. Although designed to support a pre-existing outreach volcanic hazard programme, our data suggest that the game could also be effective for improving knowledge about volcanic hazards as a stand-alone tool. The ongoing research in this study will further refine the application of serious games to volcanic hazard communication; nevertheless the use of virtual and immersive worlds for geo-gaming offers exciting opportunities to build knowledge and resilience among a diverse range of social groups within at-risk communities.
Basic information
Your information provided in this section is for research purposes and will
not be viewed by anyone other than the research team. These questions just
help the research team to understand who you are and what your role is in the
organisation of which you are a part.
Name: Organisation: Briefly describe your role: How long have you held this role?:
Outreach activities
These questions are designed to understand the current outreach you conduct relating to volcanic hazards on St. Vincent.
What volcanic hazard outreach activities does your organisation currently carry out on St. Vincent? Are you involved with public outreach programmes and in what capacity? How often are these outreach activities conducted? How long typically do these outreach activities or events last? Who are these outreach activities most often targeted for (demographic, age range, schools
etc.)? Does your organisation collaborate with other organisations to
conduct volcanic hazard outreach activities and with which organisations do you collaborate? How does your organisation follow up outreach activities? Do
they currently measure impact from such an event and how? Do you use or have you used any video games within your
current natural hazards (not exclusive to volcanic hazards) outreach activities and for what purpose?
Communication
In this section the questions relate to your experience of current methods for volcanic hazard communication.
What do you consider to be the most important information for your
organisation to communicate to the public? What methods of communication have you used?
What do you think are the main challenges in effectively communicating volcanic hazard information? To what extent do you think the general public are aware of the specific
hazards related to volcanoes and how they relate to Soufrière on a scale or
1–10? (1 How would you rate the general public's overall preparedness for
a volcanic eruption at Soufrière on a scale of 1–10? (1 From your experience, do you think there are communities or groups of
people on St Vincent that are more prepared than others and if so, why? In what ways might using video games technology as an education tool
complement or extend on the current methods of communication?
User requirements
These questions are related to what you would like or would be looking for in the finished product.
What platform do you think will be the most suitable for use in St. Vincent and why? How do you envisage a video game such as this could be integrated into volcanic hazard activities? How long do you think gameplay should last? Visualisations within the game will be created for four communities. Please identify which locations on St. Vincent you think would benefit the most from these visualisations. (Please rank with 1 being the highest priority location). During outreach sessions, is the game more likely to be played individually, in small groups or both and why?
Information to include
Are you willing to be involved in piloting the game?
What is the name of the volcano on St. Vincent? What year did this volcano last erupt? Can you name any other years when this volcano has erupted? Volcanoes have many “volcanic hazards” that occur during eruptions. What does the phrase “volcanic hazard” mean? Volcanic ash is produced by many eruptions around the world including those on St. Vincent but what exactly is volcanic ash? Where does volcanic ash come from? Lahars are common during and after volcanic eruptions. Why do lahars form? How long after a volcanic eruption can lahars occur? What is a pyroclastic flow? Why are pyroclastic flows dangerous? What colour zone of the “volcanic hazard map” are these towns in? (red, orange, yellow or green). Chateaubelair Fancy Georgetown How long can a volcanic eruption last?
The authors would like to acknowledge Plymouth University for providing funding to undertake this research. We would also like to thank the University of the West Indies Seismic Research Centre (SRC), St. Vincent National Emergency Management Organisation (NEMO), the Ministry of Education of St. Vincent and the Grenadines and the STREVA Project for supporting this research and aiding data collection. A final thank you to the editor, reviewers and Alison Stokes of Plymouth University for helping refine this manuscript. A final thank you to the editor Sam Illingworth, to the referees S. Anderson, S. Mohadjer and two anonymous reviewers and Alison Stokes of Plymouth University for helping refine this manuscript.Edited by: S. Illingworth Reviewed by: S. Anderson, S. Mohadjer and two anonymous referees