2.1 Effectiveness of risk prevention communication tools: case of the DICRIM

In this work, we focus on preventive risk communication in regulatory documents (informal communication by social networks and by word of mouth is not considered). In France, information and knowledge are communicated at a communal level using different means (see Fig. 1). The mayor establishes DICRIM intended for the public and based on information contained in DDRM (Departmental Document on Major Risks) made by the prefect. DDRM lists all essential information on major natural and technological risks at the level of their department. The municipalities required to produce a DICRIM are those in which there are Flood Risk Prevention Plans and/or those designated by prefectural decree because of their exposure to a particular major risk.

The mayor must also inform the population about major hazards at least every 2 years by way of public meetings. In addition, the safety instructions contained in the DICRIM and those which may be laid down by the operators or owners of the premises and land referred to in Article R. 125-14 of the Environmental Code are made public by means of posters. These posters are made based on a template set by the state. The mayor is also required to establish floodmarks in the municipality and carry out an inventory. They correspond to historical or exceptional new floods. They make it possible to refine the expertise of historical floods, but also to bring to life the memory of flood risk. With regard to residents, there are two distinct obligations regarding preventive information established by the Environmental Code for purchasers and tenants: (i) information on the natural hazards affecting the property and (ii) information on claims resulting from recognised natural disasters which affected all or part of the property concerned. This Purchaser Tenant Information (IAL) concerns: all lessors or sellers and all types of real estate, regardless of the destination (seasonal rentals also; MEDDE, 2013). The general public is also encouraged to establish their PFMS (family safety plan). PFMS is a booklet, provided by the state, which contains sections to filled out with relevant information by members of the public. For instance, the booklet helps them to check if they own all equipment needed in the event of a crisis.

We chose to focus on DICRIM because, in the French government's overall strategy for risk prevention, it is the main reference document in terms of informing the public about the natural and technological risks affecting the municipal territory. A brochure, film or advertisement may be just one part of a larger campaign to promote hazard readiness, and these items can and should be tested separately for efficacy and effectiveness prior to broad deployment (Sanquini et al., 2016).

In general, inequality in terms of updating the specifications, transmitting the information and controlling the execution of preventive information tools is observed and this concerns also the DICRIM. For example, in the Provence-Alpes-Côte-d'Azur (PACA) region, in December 2016 only half of the concerned communities had completed their DICRIM (ORRM, 2017). Furthermore, the French government gives indications only on the general content of the document without providing any standards notably regarding the design of the document. Works and writing guides have been proposed to develop DICRIMs (Clément et al., 2012; CEPRI, 2013; IRMa, 2014) but the justifications of these recommendations are not clearly and fully presented, which leads to disparate documents in terms of content and form. Given the large number of pages, the effectiveness of some DICRIM for residents is also questionable, as is that of other DICRIMs containing a reduced number of pages, which can lead to oversimplification and significant loss of information. Finally, there is no link between quantity and compliance of DICRIM (Douvinet et al., 2013); having a DICRIM does not mean that municipalities provide good preventive information under the law (Rode, 2012).

Effectiveness of some contents in DICRIM is also questionable. For instance, the law required that DICRIMs include maps regarding each hazard threatening the given town. Those maps are used to show to the population the hazard level geographically distributed on the territory of the municipality. But those maps are often similar to those elaborated in PPRI, which is mainly used by experts, and they are not intended to be used for communicating with the general public. Flood maps are frequently seen as an information tool rather than a communication tool (Meyer et al., 2012). That is why some maps are really difficult for some people to understand, which is regrettable because it is maps which often first draw people's attention before text. Some works propose an evaluation of flood maps effectiveness notably based on interviews or eye-tracking and also give recommendations that should be used for hazard maps contained in DICRIM (Meyer et al., 2012; Fuchs et al., 2009).

These elements must be considered carefully because individual cognitive perception is reinforced or weakened by the form and content of the provided informative document; document communication is intended to be helpful and to result in the expected impact (Terpstra et al., 2009). Consequently, assessing the effectiveness (form and content) of such legislative texts and the information they contain is an indispensable challenge, especially since this issue has received very little attention and only a few works on it have been published to date (Gominet, 2007; Kellens et al., 2013).

2.2 Analysis of candidate methodologies

Evaluating effectiveness requires the development and improvement of tools and methods. Such methods currently exist in different fields of study: advertising, geography, engineering, etc. They apply to different systems and are generally divided into three categories, according to if they (i)  involve formal population samples, (ii) do not involve formal population samples but rely on population perception assessments and (iii)  are based on systemic analytical methods that do not involve the population.

Regarding communication and marketing, advertising is one of the major tools used by companies to disseminate “persuasive” information on their markets, to inform, persuade and remind (McArthur and Griffin, 1997). Methods have been developed to assess the effectiveness of advertising (pre- and post-advertising tests). These different methodological approaches based on questionnaires are of great interest. Current studies performed to evaluate the effectiveness of preventive risk communication (including DICRIM) for the public are usually conducted using the same methodological tools: surveys (either before or after dissemination of the document) by questionnaires. Gominet (2010) and Duaut and Luneau (2008), for example, obtained their results through prior surveys of mountain communities subject to natural hazards. In the same vein, the French Minister of Agriculture conducted a series of interviews with various stakeholders on subjects including risk, to measure the effectiveness of DICRIMs or to at least obtain their views on these documents (MAAF and MEDDE, 2013). Moreover, communicating on risk can be considered as pertaining to “advertising” on how to protect oneself. As stated above, the challenge is to make the public aware of risk, get them to accept it and protect themselves by taking appropriate actions (Renn, 2014). The risk communication documents must therefore have a persuasive effect on the population in the same way as advertising seeks to convince a consumer. If the questionnaires are well constructed and the results interpreted rigourously, these methods allow for predicting the impact communication should have with regard to the pre-test and make it possible to modify the content prior to publication if necessary. In the post-test phase, they also allow for obtaining feedback on communication, on what worked or not, and on any dysfunctions in order to consider possible corrections and a new communication campaign. However, these methods also have limitations. Indeed, they require building a representative sample of the population that may require a budget and human and material resources which can be substantial. Moreover, these methods are focused on opinions and do not take into account factors that omit individuals' feelings.

Some methods do not require the use of a representative sample of the population but need indirect intervention:

• advertising efficiency standards for establishing performance comparisons, without it being necessary to use population samples;

• “connected” methods to evaluate the effectiveness of advertising such as the association of flyers, websites and/or mobile applications (QR code, website audience, image recognition, etc.). For instance, a flyer can be effective if it has a QR code. It is then possible to quantify the number of “flashes” generated by people with their smartphones to visit the associated websites. The higher the number of visits, the more effective the flyer is deemed to be;

• ROI detection elements (return on investment; click rates, visits, cookies, etc.) related to digital advertising and tracking capabilities. They allow, for example, making post-tests a certain time after the broadcast of a message to identify individuals previously exposed to it.

Connected methods can be useful in that they will help to determine whether a risk communication document aroused the interest of people by convincing them, for example, to go to the associated websites (using QR code) for additional information after reading the document. These evaluation methods also appear relevant to our case study mainly because they do not require the intervention of inhabitants to assess effectiveness. However, there is an indirect need for some people to be involved by clicking, using a QR code, etc. If it can give an idea of the effectiveness of risk communication documents, it does not provide information about the impact of its reading. Is the information really received? Although relevant in some aspects, these methods can be expensive.

Finally, the use of systemic analytical methods based on structural, functional and dysfunctional analysis of the system under study seems relevant for the analysis. Their results could be used in the next works to build an effectiveness evaluation model based on aggregated formalised indicators. Constructing an effective model will, of course, require involving the public but this will not be necessary when municipalities for instance come to use the model. In such a sense, it could provide financial gain for them. Effectiveness is the level of achievement of planned activities and achievement of expected results (Association Française de Normalisation, 2005). It refers to the concepts of function and dysfunction. Thus analytical methods, such as dependability analysis used in engineering, seem relevant. DICRIMs, and other risk communication documents, are subject to complex processes involving interdisciplinary concepts (information processing, cognition, communication, etc.). The lack of a formal, detailed framework (form and content), and lack of familiarity by the public make their effectiveness difficult to assess without a systemic and rational approach. Dependability methods allow the identification of risks and analysis of behaviour and failures. These methods are qualitative (preliminary analysis of hazards; failure modes and effects analysis, FMEA; summarised breakdowns combinations method) and quantitative (fault tree method, event tree method, etc.), based on the construction of state graphs (space states method, stochastic Petri nets), and on simulation (Monte Carlo simulation). They were developed for complex industrial systems and applied in different domains (Peyras et al., 2006; Zouakia et al., 1999; Bambara et al., 2015). These methods are interesting as they enable identifying the elements involved in the effectiveness of the system under study, determining the causes and effects of dysfunctions and listing detection elements able to pinpoint the occurrence of dysfunctions. Our work takes this direction and aims it at applying a function and dysfunction analysis method to a risk communication document. The problem to overcome is that the methods chosen must be adapted to our context, which is not one of engineering, but one of risk prevention. The first successful application of dependability methods in such a context was carried out in a field that interests us: the diagnosis of contingency plan failures (Girard, 2014; Piatyszek and Karagiannis, 2012; Bambara et al., 2015). In other words, the systemic analysis of the risk communication documents such as DICRIM can provide the means for performing qualitative analysis of effectiveness. Previous work evaluating the effectiveness of a DICRIM is based on ad hoc surveys carried out at the level of given municipalities or territories, on a given document. They also need significant human and material resources. This work differs from previous work because it allows us to obtain a generic approach based on systemic and analytical reasoning applicable to any risk communication document.

Figure 2Global methodology proposed and followed to build the effectiveness evaluation model: detailed steps leading to the detection elements. Dotted boxes show future works.

Download

4.1 System definition and needs satisfied by the system: the DICRIM

First, it is necessary to define the system to be studied precisely. Limits are defined, which then lead to considering the interactions with external environments. The system therefore takes into account the document (DICRIM) with its content, form and accessibility as described by the relevant law.

The main headings and information to be included in the DICRIM are listed in the National Model for the application of the Environment Code (Articles L 125-2 and R 125-9 to R 125-27) issued by the MTES (Ministry of Ecological and Inclusive Transition; MEDDE, 2013). Our analysis will be based on it. It is a framework with general information that must be used both by mayor and prefect to create respectively DICRIM and DDRM. To do so they have to use this framework and to add specific information, for instance information regarding the municipal territory for mayor. The major risks that must be dealt with in the DICRIM (according to their occurrence in the given town) are listed in the National Model and are as follows: floods, earthquakes, ground movements, forest fires, avalanches, storms/cyclones, volcanic eruptions, dam failures, nuclear accidents, industrial accidents and mining risks. There is no legal obligation regarding the document's format. The DICRIM can be viewed by the public at the town hall as a paper document. In some towns it is also distributed directly to residents' mailboxes or posted on the municipal website in digitised form or more rarely, in interactive format.

EFA is used for translating the need satisfied by the system. In general, this can be formalised into three questions applied here to the DICRIM: (1) to whom is the system dedicated? To the general public; (2) on what does it act? On its knowledge and risk perception; (3) why is this action necessary? (that is to say, for what purpose does the system exist?) To provide notice on hazards and on how to act when phenomena occur.

Figure 3Functional block diagram for DICRIM.

Download

4.2 External circles

It is then necessary to determine the external circles that interact with the system. They are made up of human elements, natural elements or part of other systems which can act on the document or be subject to its actions. The inventory of these external circles (designer, director, regulations, residential accommodation, etc.) is established by examining the environment of the DICRIM. We also highlight interactions between these areas and the DICRIM. Figure 3 illustrates the design, construction and operation process (reading and use by the public). Interactions between the system and its environment are visualised using a functional block diagram, in which we distinguish between contact relationships (represented by straight segments) and flows (represented by arcs). Interactions between an external circle and DICRIM identify requirements to be considered. Flow relationships between two external circles identify the service functions performed by the system. The quality of the external circles and the analysis of interactions determine the completeness of the functions obtained.

4.3 Service function and requirements

Service functions reflect the objective of a system. They are determined through relations between two external environments. Figure 3 shows that the transmitter (developer and designer) is linked with the receiver through the document. This is the process involved in preventive information on hazards. With the DICRIM, the receiver, that is to say the public, is also related to major events when they happen. They must implement what they learn from the document in terms of behaviour. Two DICRIM Service functions are then deduced:

1. Service Function 1 (SF1): the DICRIM provides information on maintaining and nurturing risk culture (learning in the operational phase). We consider that risk culture is driven by four main elements: risk awareness, knowledge, acceptance and memory (Johnson, 1993; Terpstra, 2011).

2. Service Function 2 (SF2): the DICRIM provides information on acting appropriately when facing a major event (decision-making in the operational phase). An appropriate action is based on the way the event is detected and good practices (Bostrom and Löfstedt, 2003).

These are functions fulfilled by the document to satisfy the requirements exerted on it by external circles. Constraints are obtained by examining the external circles in interaction with the document. They are listed in Table 1 (extract).

Table 1Extract of the table showing requirements features highlighted by the interactions between the DICRIM and external circles.

Download Print Version | Download XLSX

As shown by Table 1, requirements can be classified into four main groups: regulatory obligations (“compliance with laws”, “in line with the National Model given by MTES”, etc.), content requirements (“provide information on major phenomena”,), form requirements (“to be correctly designed”, etc.), and information flow (“have been received”, “have been kept”, etc.). Three of these groups, regulatory, content and form, will be kept to analyse the requirements at the component level (compliance and internal functional analysis). At this scale, we will not consider the fourth group (information flow) because it does not concern the component level. It is taken into account at a higher level of granularity when the global system is considered.

7.1 Component/service functions

Regulatory components identified during the structural analysis (see Sect. 5) are listed in Table 3 according to whether they concern the first service function of the system (the global DICRIM) or the second, identified in Sect. 4.

Table 4List of technical functions identified for both service functions of DICRIM.

Download Print Version | Download XLSX

Figure 4Role of the entities of the method.

Download

Some components are specific to one or the other service function. For instance, components 1 to 5 are specific to SF1, while components 15 and 16 are specific to SF2. Indeed, they correspond to emergency phone numbers and equipment, two elements used in case of an event to act appropriately. Some components also perform both service functions. This is the case, for instance, of “policy and protection actions” (Cp7) because they take part in fostering the acceptance of risk, which is one of the elements of risk culture characterising SF1. Indeed, protection actions include the explanation of departmental rescue plans, for example, which make people realise that the risk must be taken seriously. CP7 also performs SF2 insofar as it also gives information on the way the public will be alerted if an adverse event occurs. Alerting is one of the factors that characterise SF2. Finally, all the components are linked to at least one service function.

7.2 Service functions/technical functions

Each component has one or more so-called technical function. They contribute to the service functions. Moreover, they must satisfy the requirements. The technical functions are now linked to the different components: a component fulfils one or more technical functions (Fig. 4).

When establishing the two service functions of the DICRIM, some elements were identified as being linked to both of them. Here, these six identified elements are expressed in terms of six technical functions: four contribute to risk culture and two concern actions during an event. The technical functions of all the components are listed in Table 4 according to whether they concern the first service function or the second.

Table 5Membership of each technical function identified for the 16 components (TF1: provide information to raise awareness of risk; TF2: provide information to drive knowledge on risk; TF3: provide information to foster the acceptance of risk; TF4: provide information to maintain risk memory; TF5: provide information on event detection; TF6: make known the appropriate responses to adopt if a phenomenon occurs).

Download Print Version | Download XLSX

Figure 5Factors influencing a decision-making process related to the occurrence of a phenomenon.

Download

The identification of the technical functions is based on the following elements. The “culture of risk” is not to be seen as a more or less distributed capital but rather as a pragmatic relationship to the danger that is constructed and rebuilt perpetually, sometimes individually, sometimes collectively (CEPRI, 2013). It begins with risk awareness. Knowing of the flood does not imply feeling directly affected by this risk. This awareness of risk has a subjective dimension, specific to each individual or group (leads to define TF1; Defossez, 2012). Conversely, the awareness of what constitutes risk to oneself or to a group cannot be effective without some knowledge of this risk. This knowledge, theoretical and practical and corresponding to TF2, is also built up over time, in particular by means of information received formally or informally, such as the reception of a DICRIM for example. Knowledge and awareness about risk need to be maintained in time, so that it can be forgotten. For a variety of reasons, the transmission of generational knowledge transfer has gradually been extinguished. New populations from urban areas and tourists are often unaware of the risk their municipality is exposed to. Moreover, a phenomenon may possess a very long return period, even leading long-settled populations to forget the danger still present CEPRI (2013). This leads to TF4, characterising memory of risk. To expect an appropriated behaviour, acceptance of risk is also crucial (TF3). Again, this process is strongly related to the nature of the risk communication that has been carried out, including the credibility of the source. When an individual feels vulnerable to risk, deprived and without means to cope with risk, they may have more difficulty accepting the risk and tend to take refuge in the denial of that risk. Denial is part of a set of perception biases (overconfidence, anchoring effect, etc.), which can occur in the face of risk, running counter to the ingredients described above and which build the culture of risk (awareness, knowledge, memory, acceptance; Fig. 5; Mary and Wildavsky, 1987).

In many cases, cognitive bias takes precedence over the rationality of a decision (Slovic, 1987). For example, over-confidence corresponds to an overestimation of ones knowledge or an underestimation of ones uncertainty (Kahneman and Tversky, 1979). Most of the time subjects underestimate what they do not know. Preventive risk information systems such as the DICRIM therefore exist with a view to avoiding, as much as possible, bias taking precedence in a decision-making context. The DICRIM would therefore contribute to the risk culture by deepening these major ingredients by using technical functions. The DICRIM also provides information on acting appropriately when facing a major event. An appropriate action is based on the way the event is detected (leading to TF5) and good practices characterising TF6 (Bostrom and Löfstedt, 2003).

Table 6An extract of some content requirements satisfied by each component.

Download Print Version | Download XLSX

Table 5 represents the membership of each technical function identified for the 16 DICRIM components. We consider that components 5 to 10 generally concern each hazard. Generally, all the functions are covered by at least five components and all the components perform at least one function. TF1 is filled with 10 components. This is the function performed by most of the components. Cp14 satisfies all the technical functions of both service functions. Components 8, 9, 15 and 16 perform only one function (TF6). The pairs of components 1/2, 3/4 and 12/13 fulfil the same functions. Cp5 performs all the technical functions linked to SF1 and Cp11 the two technical functions of SF2.

The redundancy in this table is obvious. However, hasty conclusions must not be drawn from these observations. For example, although two components fulfil a single function, this does not mean that one duplicates another. Depending on the context, accomplishing the same function through several components may be beneficial. For example, awareness may not be immediate and require the addition of more information. Conversely, too much knowledge can cause boredom for the reader who may lose interest in the topic or the whole component. A detailed analysis will be performed when identifying adjustments to the content of the regulatory document. This concerns a work in progress, not presented here, that consists of interviewing the population about their perception of the DICRIM.

Table 7Functional analysis table for Component 2.

Download Print Version | Download XLSX

Table 8Example of an indicator grid for evaluating the length of the editorial component.

Download Print Version

Figure 6Diagram with a bow tie showing FMEA for Component 2.

Download

7.3 Constraints satisfied by the components

To fulfil its technical function, a component must conform to one or more requirements (see Fig. 4). The respect for the requirements by components are evaluated by detection elements that will be showed later in the article. The first requirements are regulatory ones but during our analysis we found that these requirements were not sufficient when the aim is to evaluate the effectiveness of the DICRIM. Other requirements were identified in the EFA when listing the requirements exerted on the DICRIM by external circles. The same groups must be considered here regarding the scale of the components. They are content requirements and form requirements identified in the literature (communication, advertising, etc.). It was necessary to add form requirements because the National Model does not include any specific instruction on this part, though it plays a crucial role in the effectiveness of a communication. It was also necessary to add content requirements because it remains essential to include details for effectiveness not covered by the regulatory requirements. Table 6 shows an extract of requirements filled by each component relative to content characteristics. Table 6 is composed of a total of nine columns. Each component satisfies at least one constraint. The first three DICRIM components must persuade the reader to continue reading the DICRIM because they are usually the first items they see when they open the document. There are, however, no regulations on these three components. The designer is therefore free to develop the DICRIM. Each component must capture attention to be read. The same type of table was produced for form requirements (six columns).

7.4 Functional analysis table

For each component, the technical functions along with the service function they refer to and the requirements are then gathered in functional analysis tables. Table 7, showing the component “editorial with a word from the mayor”, is an example.

Figure 7Example of the functioning of the future effectiveness model by indicator aggregation.

Download