Influence of expertise on rockfall hazard assessment using empirical methods

To date, many rockfall hazard assessment methods still consider qualitative observations within their analysis. Based on this statement, knowledge and expertise are supposed to be major parameters of rockfall assessment. To test this hypothesis, an experiment was carried out in order to evaluate the influence of knowledge and expertise on rockfall hazard assessment. Three populations were selected, having different levels of expertise: (1) students in geosciences, (2) researchers in geosciences and (3) confirmed experts. These three populations evaluated the rockfall hazard level on the same site, considering two different methods: the Laboratoire des Ponts et Chaussées (LPC) method and a method partly based on the “slope mass rating” (SMR) method. To complement the analysis, the completion of an “a priori” assessment of the rockfall hazard was requested of each population, without using any method. The LPC method is the most widely used method in France for official hazard mapping. It combines two main indicators: the predisposition to instability and the expected magnitude. Reversely, the SMR method was used as an ad hoc quantitative method to investigate the effect of quantification within a method. These procedures were applied on a test site divided into three different sectors. A statistical treatment of the results (descriptive statistical analysis, chi-square independent test and ANOVA) shows that there is a significant influence of the method used on the rockfall hazard assessment, whatever the sector. However, there is a non-significant influence of the level of expertise of the population the sectors 2 and 3. On sector 1, there is a significant influence of the level of expertise, explained by the importance of the temporal probability assessment in the rockfall hazard assessment process. The SMR-based method seems highly sensitive to the “site activity” indicator and exhibits an important dispersion in its results. However, the results are more similar with the LPC qualitative method, even in the case of sector 1.


Introduction
Rockfall instabilities are a major hazard for people, human activities and infrastructure (Bell and Glade, 2004;Moreiras, 2006).It is thus essential to assess the rockfall hazard in areas over which they are likely to occur and to propose a mapping of the hazard to manage the risk in urban areas.Natural hazard zoning has been introduced in many countries all over the world.For instance, the PPR (Plans de Prevention des Risques Naturels Previsibles) in France (Besson et al., 1999) and the cartes de dangers in Switzerland (Leroi et al., 2005) evaluate the hazard level in affected zones, according to a predefined set of hazard classes.Based on this hazard maps, different areas are identified: areas where construction is restricted, areas where a monitoring system or a protection system is required for reducing the risk and areas where no restrictions apply for constructions (Fell et al., 2008).Because of the implications in the territory management, the rockfall assessment must be as accurate as possible (neither underestimated nor overestimated) and reliable.
Rockfall hazard can be defined as the probability that a specific location at the toe of a studied slope will be reached by a rockfall of a given magnitude (Jaboyedoff et al., 2001).Whatever the magnitude of the expected rockfall, the probability can be divided into two terms: the failure probability and the propagation probability (Jaboyedoff et al., 2005).
Various methods simulate the trajectory of rock masses fragments after rupture and evaluate the propagation probability (Dorren, 2003).However, to date, there is no fully reliable method to estimate the failure probability (Hantz, 2007).Even though some existing methods are quantitative and based on historical inventory (Dussauge-Peisser et al., 2002;Hungr et al., 1999;Luckman, 1976), in most cases such an inventory is not available.Then, qualitative assessment methods are used to assess the rockfall probability failure.These methods are mainly based on expert judgment (Abella and Van Westen, 2008;Budetta, 2004;Effendiantz et al., 2004;Hantz et al., 2003;Jaboyedoff et al., 2001).Therefore, experts in charge of the hazard assessment have a key role.The method considered could likewise have an influence in the hazard assessment.Different research works (Abbruzzese and Labiouse, 2010;Bormioli et al., 2011) have compared different mapping methods and evaluated the influence of the method chosen on the obtained results.These studies highlighted the statistical significance of the chosen method: the use of one method over another gives different levels of failure probability for the same sector.
In this paper, an experiment is undertaken in order to evaluate the influence of the method and the level of expertise on the rockfall failure probability assessment.To simplify the reading, and the understanding of the paper, the term "rockfall hazard" will be used when referring to the rockfall failure probability.It corresponds to the combination of the occurrence probability, the temporal probability and the magnitude (volume).The rockfall hazard is assessed on a test site by three populations with different levels of expertise.Two different methods for rockfall hazard assessment are used: (i) a qualitative one, the Laboratoire des Ponts et Chaussées (LPC) method, which is the method mainly used in France, and (ii) a quantitative one, based on the slope mass rating (SMR).The experiment was conducted in two phases: the first one was realized in May 2012 and the preliminary results were presented in EUROCK Congress (Delonca et al., 2013).The second one was realized in May 2013, in order to confirm the first results, by increasing the size of the three populations and adding statistical procedure to the study.For each phase, the same protocol has been followed: three populations with different levels of expertise have assessed the level of rockfall hazard by considering two different methods of analysis, on three sectors of the same test site.The experimental protocol considered in the study is presented in more details in the first part of the paper.Then, the results are presented and discussed for the two phases together.
The objective of the study is to consider different methods of rockfall hazard assessment and to evaluate the differences in terms of levels of rockfall hazard.The levels of rockfall hazard considered in the study correspond to classical rockfall hazard levels: very low, low, moderate and high (Copons et al, 2008;Bauer, 2011;OFAT, 1997).These levels are commonly used to build hazard maps for risk management in urban areas.Thus, it is possible to compare these levels, which are obtained using different methods.Moreover, the influence of the level of expertise on the result, and so on the obtained hazard levels, is also investigated.Indeed, it may interest engineers and researchers in charge of hazard mapping or those concerned with the development of rockfall assessment methods.Note that the objective of the paper is not to evaluate the "true" level of rockfall hazard but to compare the evaluation process, considering different levels of expertise and different methods.

Selection of the test site
The experiment was realized on a cliff situated in Liverdun, a town close to Nancy in France.The test site has been chosen because of its history.It is an old quarry that became a climbing site, which has been closed after repetitive rockfalls.Moreover, it has been previously classified as presenting a high-level rockfall hazard (Moiriat et al., 2008).The methodology used to evaluate the rockfall hazard on the site is the following: (1) bibliographic review of all the documents available and (2) field recognitions.The first step leads to evaluate the risk area at a departmental scale.The second step leads to prioritize the level of rockfall hazard on the area of study.Therefore, the evaluation is made at a regional scale.The site is a 50 m long cliff of Jurassic limestone that is made up of massive blocks at the base and small blocks in the upper part (Delonca et al., 2013).
Three sectors were identified (Delonca et al., 2013): -Sector 1 presents a massive wall with well-defined stratification beds.A pluri-decametric fracture isolates a rock panel several hundreds of cubic meters in size.At the top, small unstable blocks are present (Fig. 1a).
-In sector 2, two major fractures form a rock wedge of a few cubic meters in size.Small unstable blocks are present in the upper part (Fig. 1b).
-In sector 3, limestone beds are overhanging, and some blocks at the top are unstable (Fig. 2).
The prevailing failure mechanism on the three sectors is described in the following (after Hantz et al., 2003): for sector 1, the main failure mechanism is column toppling; for sector 2, it is wedge slide; for sector 3, it is overhang failure.The latter type of failure is also present on the two others sectors.The overhang failure is the result of a traction failure for some of the small blocks at the top of the three sectors.Moreover, there also exists a failure mechanism that involves shearing of joints on the three studied sectors.

Population
Three populations were involved in the experiment.
-A total of 38 first-year MSc students in the geosciences, confronted with a hazard assessment study for the first time.These students may later be asked to prepare hazard or risk maps, for example during an internship.
-A total of 10 researchers in the geosciences, working in the field of hazard and risk assessment but not accustomed to regular rockfall assessment studies.
-A total of 8 confirmed experts benefiting from a long experience in rockfall hazard assessment and risk studies.
The comparison of the rockfall hazard evaluated by each population is supposed to highlight the influence of the level of expertise on the rockfall hazard assessment.Note that the expert assessment is not necessarily the "best" one or the "true" one.

Methods used for the rockfall hazard assessment
At first, an a priori assessment was requested from each population, without making use of any method.Then, two rockfall hazard assessment methods are used: the LPC method, which is a qualitative method frequently used in France, and a SMR-based method that relies on more quantitative information.

A priori assessment
The first assessment of the rockfall hazard was requested from each individual of the three populations.Four failure probability levels were proposed: (i) zero, (ii) low, (iii) medium and (iv) high.A presentation of rockfall hazard theory and a presentation of the methods and their parameters were delivered before this work as well as a document containing additional information on the history of the site.The objective of this first assessment was to compare it with the assessment later carried out using the two other methods and to estimate the differences between an a priori assessment and one based on the use of a guided method.

LPC method
The LPC method is detailed in Laboratoire des Ponts et Chaussées (Effendiantz et al., 2004).It is a qualitative method that is frequently used in France for official hazard mapping.It consists of two main steps: (1) the collection of preliminary data and (2) the use of these data to localize and characterize the potential instability.
The collected preliminary data incorporate all relevant information regarding the study site, including the following.
-Documentary information: the objective is to exploit the archives to avoid duplication of studies already realized.
-Historical information: to study the past of the site to determine whether events have already occurred on the site.
-Geological information: to analyze the lithological and stratigraphic data and the regional geological history to establish the geological context of the study site.
-Structural information: to identify the structural characteristics of the study site on different scales and to propose global and local structural models.
-Morphological information: to identify the main historical steps leading to the actual morphology.
-Hydrogeological, hydrological and climatic information: to characterize the fluid intakes, their nature and their importance.It is also to identify the flows inside the massif and unfavorable climate patterns (e.g., freezing, thawing and important thermal contrasts).-Information regarding vegetation: to identify and characterize the main vegetation on the site and its influence on stability or instability processes.
-Potential mechanism of rupture: to identify the potential mechanisms of rupture associated with the studied zone.
-Sectoring: to identify homogeneous areas from the hazard characterization point of view.This step is performed in the study of large zones.
Following the collection of these preliminary data, the potential instabilities are localized and characterized.To do so, three new stages are carried out.
i. Localize and identify potential instabilities.In this stage, the expert in charge of the study observes the following parameters: 1. Nature of the studied area: description of the instable rock mass, lithology, hydraulic and hydrologic data.
2. Geometric parameters: geometry and dimensions of the potentially unstable compartments.
4. Triggering factors: e.g., interstitial water, rainfall, high temperature variation, freeze-thaw cycle and vegetation This stage helps to determine the unstable volume (magnitude) and the potential mechanism of failure.In the experiment proposed in this paper, these parameters were coded to process them statistically (Fig. 3).This codification, which has been validated by expert users of the LPC method for the purposes of this experiment, does not belong to the original method.
ii. Define the rockfall hazard.The analysis of the previous parameters helps to define potentially unstable volumes and the potential mechanism of rupture associated with these volumes.For each of these, a coupled "temporal probability / occurrence probability" is assessed.These two terms are qualitatively defined as follows.
-The occurrence probability is subjectively assessed from the parameters presented in Fig. 3.It answers the question, "Can the rockfall occur?"The qualitative scale of the level of occurrence probability is presented in Table 1.-The temporal probability corresponds to the annual frequency or return period of the rockfall on the study site.It is defined as the time expected to failure.In other words, the temporal probability is defined by the probability that the failure occurs before an expected delay.The periods are defined on a scale from "imminent" to "long term".The definitions of the terms are presented in Table 2.
Then, for each volume under consideration, these two probabilities are combined in a matrix (Table 3).The plotted couples are then used to provide the best response to the risk: quick action for the shortest terms and the strongest issues and the planning of preventive actions for the longest terms.
iii.Qualification of the probability of propagation.With the LPC method, it is proposed to carry out a qualitative  (2004).
Very high The occurrence of the phenomenon is normal.Its non-occurrence will be exceptional.

High
The occurrence of the phenomenon is more probable than its non-occurrence.Moderate The occurrence of the phenomenon is equivalent to its non-occurrence.

Low
The non-occurrence of the phenomenon is more probable than its occurrence.

Very low
The non-occurrence of the phenomenon is normal.Its occurrence will be exceptional.trajectory analysis before considering a more advanced study using numerical simulations.We do not detail this last step because it has not been considered in the experiment.
The use of the temporal probability / occurrence probability matrix is not easy for beginners.Therefore, we have proposed a new matrix, developed with experts, to assess the rockfall hazard level.The temporal probability and the occurrence probability are combined to assess the level of predisposition to instability (Table 4).Finally, the predisposition to instability is coupled with the volume (magnitude of the phenomenon) to determine the rockfall hazard (Table 5).Three levels of rockfall hazard are defined: low, medium and high.
Several volumes can be identified on a single sector, and then several values for the level of rockfall hazard can be assessed.
The global rockfall hazard level of the site corresponds to the worst hazard level on the study site.

SMR-based method
The SMR index, proposed by Romana (1985), is a geomechanical classification commonly used for the characterization of rock slopes (Corominas Dulcet and Mavrouli, 2009;Irigaray et al., 2003) and derived from the rock mass rating (RMR) as follows: where RMR b is the basic RMR index resulting from Bieniawski's rock mass classification without any correction (Bieniawski, 1972).It is obtained by adding rating values for the following five parameters.
-The strength of the intact rock.
-The rock quality designation (RQD) (Deere and Miller, 1966), which gives a quantitative estimate of the rock mass fracturing based on the study of cores obtained by drilling.The RQD is defined as the percentage of intact pieces of length greater than 10 cm over the total length of the hole.It can also be estimated from surface measurements.
-The spacing of discontinuities.
-The condition of discontinuities: the roughness, weathering and opening of the discontinuities are assessed.
-The water inflow through discontinuities and/or the pore pressure ratio.
F 1 , F 2 , F 3 and F 4 are defined as follows: -F 1 characterizes the angle (A) between the slope face strike and joint azimuth.It ranges from 0.15 to 1.00, according to the relationship F 1 = (1 − sin(A)) 2 .A value of 1 indicates that the joint azimuth and face strike are parallel.
-F 2 refers to the joint dip angle.For a plane sliding mechanism, its value ranges from 0.15 to 1, according to the relationship F 2 = tan(B j ) 2 , where B j denotes the joint dip angle.For a toppling mode of failure, F 2 is equal to 1.00.
-F 3 reflects the relationship between the slope and joint dips.This parameter uses the Bieniawski adjustment factors that range from −60 to 0 points and reflects the probability that outcropping blocks will be subjected to planar and wedge failure mechanisms.
-F 4 is an adjustment factor to take into account the method of excavation (natural slope, presplitting, smooth blasting, blasting or mechanical and deficient blasting).Its value ranges from −8 to 15, and it is chosen empirically.
The SMR calculation leads to five stability classes (Romana, 1985).Then, based on the work of El-Shayeb et al. (1997) or El-Shayeb (1999), a level of the site activity Very High The fall of the rock mass will happen imminently.

High
The fall of the rock mass is more probable than its stability.The temporal probability estimate is approximately 2 years.

Moderate
The probability of the fall of the rock mass is equivalent to its stability.The temporal probability estimate is approximately 10 years.

Low
The stability of the rock mass is more probable than its fall.The temporal probability estimate is approximately 30-50 years.

Very Low
The fall of the rock mass will be exceptional, or the temporal probability estimate is approximately 100-150 years.
is evaluated.There are four different levels of activity corresponding to the following.
1. Sleeping: weathering traces are faded, and there is no alteration of the rock mass.
3. Low active: recent weathering traces and deeper rock mass alteration.
4. Active: numerous weathering traces and deep rock mass alteration.
The combination of the SMR class and the site activity provides a level of predisposition to the instability (Table 6), which is coupled to the volume of unstable masses to obtain the rockfall hazard level (Table 7).
Note that the level of activity, assessed considering the weathering traces is considered as the history of the site (El-Shayeb et al, 1997).Thus, it includes the temporal activity of rockfalls.Indeed, it is considered that some weathering traces can be used to evaluate the imminent and very shortterm qualitative scale of the level of temporal probability (Table 2 -LPC method).Thus, even though this approach is less global than the one proposed by the LPC, it allows the rockfall hazard level to be evaluated.The different terms used in both the LPC and SMR-based methods are presented in Fig. 4.

Complementary data
The on-site observations took place at the foot of the cliff.Each individual had a compass, a sclerometer and a geologist's hammer with him and had to report his observations and measurements on a separate form for each sector.On this form, each method was presented again, and all the used parameters were listed (cf.Fig. 3 for the LPC method).

Results
To statistically treat the results, the level of the rockfall hazard was coded as follows.
This coding allows the level of rockfall hazard to be quantified and reflects its intuitive increase.Other encodings have been tested, as discussed in Sect.4.3.

Methods for the statistical treatment
Each individual had to make three types of rockfall hazard assessment: an a priori assessment, an assessment with the LPC method and an assessment with the SMR-based method.After the experiments were carried out, a descriptive statistical analysis was performed to compare the levels of the rockfall hazard assessment.Then, the results were analyzed using the chi-square independence test and the analysis of variance (ANOVA) method (Scheffé, 1959) to assess the significance of the influence of the level of expertise and of the influence of the method used in determining the rockfall hazard level.
Finally, the means of the LPC and SMR-based method parameters were compared to identify those that most influence the rockfall hazard assessment.

Preliminary descriptive analysis
An initial statistical treatment has been performed for the three sectors.Figure 5 and Table 8 show the results and exhibit the following.
-Regarding the a priori assessment, there is a wide dispersion (standard deviation higher than 0.5) of the results in terms of rockfall hazard levels for all populations in all sectors, except for the experts in sector 3. Except for sector 3, where the populations give similar results, the students more often give a higher value for the rockfall hazard level, followed by the experts and finally the researchers.This can be explained by the level of inexperience of the students, who were initially impressed by the presence of cracks and overhanging blocks on the cliff.
-Regarding the LPC method, the results are similar for all population groups, with less dispersion than with the a priori assessment.-With the SMR-based method, the results exhibit more dispersion for the three sectors, with the standard deviations being higher than 0.5 except for the students in sector 2. We also note that the mean rockfall hazard level is always lower than that from the LPC method.
For comparison, a random sampling on the SMR matrix (predisposition to instability -magnitude) gives the following results (number of favorable cases on number of possible cases): a 6 / 16 = 37.5 % low-rockfall hazard level, 7 / 16 = 43.75% moderate-rockfall hazard level and 3 / 16 = 18.75 % high-rockfall hazard level, corresponding to a mean value of 1.8.Most of the values obtained in the experiment are higher than what would be randomly obtained.
The influence of the level of expertise and of the chosen method on the level of rockfall hazard could not be clearly determined from this preliminary descriptive analysis.This is the reason why statistical tests have also been performed.

Influence of the level of expertise and the method
on the level of rockfall hazard The chi-square independence test allows the dependence between two qualitative variables to be investigated.Let x 1 ,. . ., x i ,. . ., x p and y 1 ,. . ., y j ,. . ., y q be the terms (categories) of two qualitative variables X and Y .A sample  of n individuals from whom the values of the two variables were simultaneously taken yielded the following results: n ij is the number of individuals who presented both the x i value of X and the y j value of Y .n i. and n. j are, respectively, the total of line x i and the total of column y j .It is then possible to build the contingency table of the observed values (Table 9).Under the hypothesis that variables X and Y are independent, we can also build a contingency table of theoretical values equal to n i. •n.j n at the intersection of row i and column j .It is then possible to calculate the following quantity.
which obeys a χ 2 distribution with (p − 1)(q − 1) degrees of freedom.This test is performed side by side, between (i) the method and the level of rockfall hazard (here considered as a categorical variable) and (ii) the level of expertise and the level of rockfall hazard.As an example, Tables 10 and 11 present the contingency tables of the observed (o ij ) and theoretical values (t ij ), respectively, for the chi-square test to be performed between the level of expertise (variable X) and the level of rockfall hazard (variable Y ) for sector 1.The χ 2 distance is then calculated following Eq.( 2): Table 12 shows the distances computed for the twoindependency test carried out on each sector together with their p values.A p value lower than 0.05 indicates that the independency hypothesis can be rejected at a 5 % risk level.This table highlights the following: a non-dependency between the level of expertise and the level of rockfall hazard for sectors 2 and 3;  a dependency between the level of expertise and the level of rockfall hazard for sector 1; a dependency between the method and the level of rockfall hazard for the three sectors.

Analysis of variance
ANOVA is a statistical test indicating the influence of qualitative variable(s) on a quantitative variable to be assessed.It is based on the comparison of the mean values of the quantitative variable for each category of the qualitative variable.
Before performing the ANOVA, the normality of the level of rockfall hazard (the quantitative variable) and its homoscedasticity have to be verified.The levels of rockfall hazard are discrete data, which make the condition of normality difficult to assume.However, authors such as Legendre and Borcard (2008) and Underwood (1996) state that, in a balanced experimental design, ANOVA has problems with the heterogeneity of the variance when the variance is in high contrast to others, but the ANOVA is little affected by nonnormality.This condition is considered as not discriminating to achieve the ANOVA.

A. Delonca et al.: Influence of expertise on rockfall hazard assessment using empirical methods
To test the homogeneity of the variances, there are several tests, including Levene's test, that are insensitive to the normality of the data (Legendre and Borcard, 2008).If Levene's test is statistically significant, the assumption of the homogeneity of variances should be rejected.We used the software program R to perform this test.For the three sectors, Levene's test has shown that we can consider the variances to be homogeneous.
In the ANOVA, the influence of a factor is considered significant when the p value associated with this factor is lower than a given risk such as 5 or 1 % (error chosen here).The ANOVA results are presented in Table 13.To help the reader in the analysis, the significant factors are grayed.The analysis of variance shows the following: a very significant influence of the factor "method" on the level of rockfall hazard assessments for the three sectors; a very significant influence of the factor "expertise" on the level of rockfall hazard assessments for sector 1; the non-significant influence of the factor "expertise" on the level of rockfall hazard assessments for sectors 2 and 3 at a 1 % risk (but a significant influence at a 5 % risk); the absence of interaction between these two factors on the level of rockfall hazard assessments (for all three sectors) at a 1 % risk or even at 10 % risk.
These results confirm the results given by the test of independence carried out previously.They first exhibit the main importance of the method chosen, as found previously for the three sectors.Then, the great influence of the level of expertise for sector 1 is highlighted.Finally, both tests highlight the negligible influence of the level of expertise for sectors 2 and 3, regardless of the level of confidence we are working with.

Influence of the parameters of the LPC and SMR-based methods
To better understand the influence of the LPC and SMRbased methods on the level of rockfall hazard, a comparison of the mean values of the parameters used in these methods has been conducted (Table 14).
For each parameter presented, a two-by-two comparison of the mean values has been carried out.More precisely, we have compared the mean values of the levels given for each parameter by (i) the students and the researchers, (ii) the students and the experts and (iii) the researchers and the experts.The objective was to identify the mean values that are statistically similar and those that are different.This analysis could explain the influence of the chosen method previously highlighted, depending on the sector and the level of expertise.
The first step consists of checking the hypothesis of equal variance for each couple of the population.If the variances can be considered equals, it is then possible to compare the mean values.Let n 1 and s 2 1 be the size and the variance of the first population sample (students, researchers or experts), and let be n 2 and s 2 2 be the size and the variance of the second population sample.Under the hypothesis of equal population variances, the two estimates of the variance should be equal.This hypothesis has been tested (F test) and verified, and the comparison of the means can then be performed.
Let m 1 and m 2 be, respectively, the mean values of the first and second population samples under test.Both are unbiased estimates of the population means (µ 1 and µ 2 ).Under the assumption that µ 1 = µ 2 , we compute the tratio (Eq.3), which is supposed to belong to a Student's t distribution with (n 1 + n 2 − 2) degrees of freedom.This value has then to be compared to the interval [−t α/2 t α/2 ], where α is the risk and (1 − α) the confidence level: where m 1 is the average of the first population tested (students, researchers or experts), and m 2 is the average of the second population tested; n 1 is the sample size of the first population, and n 2 is the sample size of the second population; σ * 2 is the unbiased estimate of the common variance of the two populations based on s 1 , s 2 , n 1 and n 2 .
If t belongs to the interval [−t α/2 t α/2 ], α being the chosen risk, we can conclude that the two population means are not significantly different.We considered a 5 % risk, so the interval is thus approximately equal to [−2,2].Table 15 presents the t ratio for the different parameters tested.Table 16 summarizes the results and shows the following: the predisposition to the instability (LPC method only) is the same, whatever the level of expertise and the sector; the magnitude is similarly assessed for sectors 2 and 3, regardless of the level of expertise, for both methods (LPC and SMR-based methods); the magnitude is assessed differently by the students on one side and the experts on the other side for sector 1; -The SMR classes in the SMR-based method is the same for sectors 2 and 3, but not for sector 1, where there is a significant difference between the students and the experts; the level of activity is assessed differently for the three sectors.Then, seeing that the combination of the SMR classes and the site activity provides the level of predisposition to the instability, the predisposition to the instability is assessed differently for the three sectors.Note that another analysis (principal component analysis) has been realized to compare directly the parameters used to evaluate the occurrence probability of the LPC method (Fig. 3).However, the analysis did not produce any concluding results.

Choice of hazard assessment methods
During the experiment, three methods were used to assess the rockfall hazard.First, each participant was asked to assess the level of rockfall hazard from his own experience without using any tool, form or measurement.Second, a qualitative method (LPC method) and, third, a quantitative method (SMR-based method) were used.In both of these latter cases, a form was provided to guide the process of assessment.
We chose to use the LPC method due to its wide use in France for most official hazard mapping.This methodology aims to formalize the practice gradually developed over 2 decades through many field studies produced by both operational services of the French government and local authorities.Many experts contributed to the preparation of this guide, and some of these experts also participated in our experiment.The methodology proposed by the LPC method is somehow classical and follows the following steps as previously described: 1. a literature review, which avoids repeating studies or investigations already done; 2. an historical review, which makes an initial zoning of hazard levels, taking into account past events; 3. a geomorphological analysis, during which the level of the rockfall hazard is actually assessed.
These steps can be found in many other methods that exist in different countries.In Spain, for instance, Copons and Vilaplana (2008) proposed a similar qualitative method for hazard mapping.Several authors also tried to introduce more quantification to hazard assessment for parameters that are usually qualitatively addressed, such as in the Matterock method developed in Switzerland (Baillifard et al., 2003), the RES method (Hudson, 1993;Mazzoccola and Hudson, 1996;Rozos et al., 2008) and the RHAP method (Ferrero et al., 2011;Lombardia, 2000).Then, even though the LPC method is not an internationally recognized method, we may consider that it treats the hazard analysis in a common manner, similar to other methods.In contrast, the SMR method, also used in the present work, is commonly used in rock slope stability analysis along roadways (Budetta, 2004;Corominas Dulcet and Mavrouli, 2009;Irigaray et al., 2003;Tomas et al., 2012).It is derived from the globally used RMR method for the stability study of excavated slopes or underground excavations.
Even though it is a qualitative method, the LPC method helps the user to identify the block masses that are potentially instable and the volume concerned and leads to the assessment of the level of rockfall hazard in a similar manner by the three tested populations.However, the introduction of a paper form to assist in the process of the assessment of the level of rockfall hazard could have introduced a bias favoring the LPC method in terms of less dispersion in the assessment results when compared with the results of the SMR-based method.As a result, such a form could be promoted to engineers making use of the LPC method to enforce a common interpretation of some aspects of the method.
The SMR-based method shows results similar to those obtained by the LPC method for students.However, for researchers and experts, the dispersion in the results is more important.The assessment of the site activity is not precise enough to allow the same level of activity to be assessed by all the individuals.Surprisingly, the use of a partly quantitative method does not prevent differences in the final rockfall hazard level, mainly because of the assessment of the
The terms used to describe the activity may be clarified to reduce this effect.Moreover, the prevailing failure mechanism on the three sectors plays an important role in the use of the SMR-based method.Indeed, the SMR method is routinely used to assess rockfall susceptibility of failure mechanisms involving shear on joints (including flexural toppling) though it seems inapplicable to failures caused by traction.
In the test site, the main failure mechanisms are column toppling, wedge slide and overhang failure.However, there also exists a failure mechanism that involves shearing of joints on the three studied sectors.Because it is not the main failure mechanism on the test site, the SMR-based method has been considered anyway into the analysis.

Choice of the test site
For the experiment, one test site was selected, among which three sectors could be defined.These three sectors provide repeatability in the analysis and produce more relevant results.However, because there are three sectors on the same site, all of them are composed of the same lithology.Hence, the results obtained in the present study only apply to this case study, and the experiment should be repeated on other sites with different lithology and geological structures to prove that the findings are generic.The test site was chosen because of its history, as it was a climbing site which has been closed after repetitive rockfalls.However, no quantitative information concerning the occurrence of the phenomenon was available.The site is not monitored, and the sole data considered in the analysis were the visual evidence of events.The scenario considered here is that an expert is in charge of the evaluation of a new site, and he does not have any temporal data available to him.Therefore, to propose a more complete analysis, it would be interesting to consider another test site where the estimation of temporal frequency is possible.A national French project is underway that may provide a framework to repeat this experiment on different study sites, with more students, researchers and experts involved, as well as other methods to be tested.However, no sites have yet been chosen to carry out this experiment again.

Coding of the level of rockfall hazard
As explained previously, to carry out the statistical analysis, the three rockfall hazard levels were transformed into quantitative variables: the low-rockfall hazard level was coded to 1, the moderate-rockfall hazard level to 2 and the high-rockfall hazard level to 3, so as to reflect the increase of the rockfall hazard in a familiar and convenient way.However, to confirm that this coding has no influence on the results, we performed additional ANOVAs with different encodings.The following cases were used, respectively, for low-, moderate-and highrockfall hazard levels: -0, 5 and 10; -1, 2 and 5; -1, 4 and 5; -3, 2 and 1 (converse coding).
It was found that the coding choice does not affect the conclusions regarding the significant influence of the method on all sectors and the non-significant influence of the level of expertise on two (nos. 2 and 3) of the three sectors.
Note that the quantification of the variable is used only for the statistical analysis.

Conclusions
The novel experiment developed in this paper had the goal of statistically evaluating the influence of the level of expertise and the choice of the method used for the rockfall hazard assessment.Three levels of expertise (students, researchers and experts) and two methods (LPC and SMR-based, plus an a priori assessment) were used on three sectors at one test site.
The main result obtained is that the influence of the level of expertise is not significant on the rockfall hazard assessment, which means that geoscience students and geoscience researchers who are not experienced at rockfall evaluations provide the same assessment as engineers experienced in rockfall hazard assessment.
More precisely, the qualitative analysis of the results, as well as the chi-square independence test and the ANOVA test highlighted the following: a statistically significant influence of the factor "method" on the level of the rockfall hazard assessment for all three sectors; a statistically significant influence of the factor "expertise" on the level of the rockfall hazard assessments for sector 1; a statistically non-significant influence of the factor "expertise" on the level of the rockfall hazard assessments for sectors 2 and 3.
The influence of the level of expertise on the rockfall hazard assessment for sector 1 can be explained by the characteristics of this sector.A pluri-decametric fracture isolates a rock panel several hundreds of cubic meters in size.After discussions with the three populations, it appears that this particular configuration is a challenge to the students, who tend to overestimate the rockfall hazard.For researchers and experts, the stability of the sector is not an issue in the present state, and the hazard level should be low.Here, we highlight the difficulty of assessing the temporal probability on a specific site.Moreover, the statistical analysis showed a high dispersion in the rockfall hazard assessment for the a priori assessment, especially for the students and researchers, explained by their inexperience in this type of exercise.This result, coupled with the previous one, highlights the "plus-value" of the experts: they do not overestimate the level of rockfall hazard in complex cases, and their assessment is more homogeneous and based on similar cases they have already studied due to their experience.Indeed, for sector 1, which exhibits the above mentioned pluri-decametric fracture, the results show a higher ratio of low-rockfall hazard levels for the experts.This suggests that the experts link the rockfall hazard assessment to the temporal probability assessment.For them, there is no risk of failure within a short time.
Other assessments of the rockfall hazard level will be realized on other sites to enrich this experiment and confirm or challenge the conclusions.The rockfall hazard level will be assessed by other people (students, researchers and experts), and other methods will be used.
If confirmed, the conclusions drawn in the presented paper would suggest that the use of a qualitative approach could be as relevant as a quantitative approach for rockfall hazard assessment even though it leaves room for some subjectivity.Nevertheless, the use of a standard form as proposed in this study (as exists for many other methods, either basically quantitative or qualitative) could be promoted to engineers making use of the LPC method to enforce a common interpretation of some aspects of the method.Such a form may contribute to a reduction of the subjectivity expected in a qualitative method.

Figure 4 .
Figure 4. Terms considered in the LPC and SMR-based method.

Table 1 .
Qualitative scale of the level of occurrence probability, after the guide of the Laboratoire des Ponts et Chaussées

Table 2 .
Qualitative scale of the level of temporal probability, after the guide of the Laboratoire desPonts et Chaussées (2004).

Table 3 .
Matrix used to assess the coupled "temporal probability/occurrence probability", after the guide of the Laboratoire desPonts et  Chaussées (2004).Temporal probability Imminent Very short-term Short-term Medium-term Long-term

Table 4 .
Qualitative scale of the predisposition to instability.

Table 5 .
Assessment of the rockfall hazard with the LPC method.

Table 6 .
Assessment of the predisposition to instability with the SMR method.
Students achieved the highest mean score whatever the sector.Experts and researchers are closer to each other, particularly in sectors 2 and 3.For comparison purposes, a random sampling (using uniform distribution) of the levels of predisposition to instability and magnitude provided a 6 / 25 = 24 % lowrockfall hazard level, 9 / 25 = 36 % moderate-rockfall hazard level and 10 / 25 = 40 % high-rockfall hazard level (i.e., a mean value of 2.16); the observed values are clearly different from what a random assessment would give.

Table 7 .
Assessment of the rockfall hazard with the SMR method.

Table 8 .
Mean and standard deviation of the rockfall hazard levels for the three sectors.

Table 9 .
Contingency table of the observed values of the X and Y variables.

Table 10 .
Contingency table of the observed values for the chi-square test performed between the level of expertise and the level of rockfall hazard (sector 1).Observed values (o) Low-rockfall hazard level Medium-rockfall hazard level High-rockfall hazard level Total

Table 11 .
Contingency table of the theoretical values for the chi-square test performed between the level of expertise and the level of rockfall hazard (sector 1).

Table 12 .
Chi-square distance between the observed and theoretical values for all the cases tested, under the hypothesis of independency.

Table 13 .
Results of ANOVA: influence of the level of expertise and the choice of the method on the level of rockfall hazard for the three sectors.The bold values highlight the significantly influencing factors, for which the p values are lower than 1 %.Degrees of freedom Sum of squares Mean square Fisher-Snedecor test value p value

Table 14 .
Code of the parameters used in the LPC and SMR-based methods.

Table 15 .
Values of the t ratio, to compare the average 2 by 2 of the parameters of the LPC and SMR-based methods.The bold values correspond to the population means that cannot be significantly differentiated (or can be considered as equal).

Table 16 .
Sample mean and standard deviation of the values of the LPC and SMR-based parameters.The bold values correspond to the population means that cannot be significantly differentiated.