Some STREVA researchers keep an online blog. 

Our 'Opinion' page reproduces posts that are related to STREVA. 

St Vincent Paired Comparison

Results from the STREVA Paired comparison elicitation: an initial discussion

Jenni Barclay, Willy Aspinall, Richie Robertson, Richard Herd, Anna Hicks



This was an initial attempt to elicit and analyse factors influencing two things: (1) how we might combine and weight differing dimensions of risk and (2) the uncertainty and differences in experience and views that guide thinking about these different dimensions of risk.

We used Soufrière St. Vincent to develop two eruptive scenarios. This provided a focus around which everyone could make judgements about the most important factors that contribute to (i) saving lives during an acute volcanic emergency and (ii) assisting populations to adapt and cope with the stresses imposed by any activity. St. Vincent is an island in the Lesser Antilles which perhaps would bear some similarities to the situation around the Soufrière Hills Volcano on Montserrat in the event of re-awakened activity.  Thus, we used some of the key driving factors emerging in our ongoing forensic analysis of Montserrat activity to define the categories.

Paired comparison: what is it?

A pairwise comparison survey allows experts to provide judgments on a structured ranking of a set of variables, rather than ascribe individual point values or probability distributions to them. Read more about it here.

The Construction of the St. Vincent Scenarios

Two scenarios were constructed for plausible eruptive pathways the volcano might take and likely associated observations and monitoring signals. Prior to the development of the scenarios the group heard a description of historical activity associated with Soufriere of St. Vincent.

In no sense were they the only scenarios, and are not necessarily the most likely, but they span a reasonable range of eruptive experience and styles in which one might expect Soufriere of St Vincent to behave.  Elements from the lead-up to the 1902 and 1979 SSV activity were used, but we were not explicitly reconstructing those eruptions. Read the details of the scenarios here.

For each scenario respondents were asked to produce two paired comparisons that assessed, how the elements described are most important in terms of (a) saving lives [Lives] and (b) enabling the population to adapt and cope [Livelihoods].

These elements were informed by discussion among the STREVA group about the factors that had become important at different times during the Soufrière Hills eruption:

A: Knowledge of the range of possible hazards at this volcano and their likelihood of occurrence [SciKnow.]

B: Ability to interpret precursors to changes in behaviour from monitoring data and visual observations. [Precursors]

C: Sound judgement of extent of uncertainty in hazard knowledge and/or ability to interpret behaviour  [Uncertainty]

D: Capability to communicate escalating risks and uncertainty to those impacted by activity [Risk Comms]

E: Capacity to relocate and rehouse people [Relocatability]

F: Strong social cohesion and community relationships [Social Cohesn.]

G: Good, flexible infrastructure in unaffected areas, including alternative land [Infrastruct.]

H: Ability to decide when eruption is over [EruptOver]

I:  Public participation   in decision-making processes [PubParticip.]


 Respondents were split into three groupings post-hoc:

(i) physical scientists who had regular experience of providing scientific advice to those managing volcanic activity (sci-advisor, 10 people);

(ii) physical scientists largely from a research-based background (physical scientists, 12 people)

(iii) interdisciplinary scientists (interdisciplin, 9 people). The latter group was the most heterogeneous and consisted of those engaged in social-science based research around disaster risk, resilience and vulnerability as well as those involved in funding and managing this type of research or in working with practitioners or other natural hazard fields.


Comparing the two scenarios

Figures 1 and 2 show the ellipses associated with the factor rank scores and variance for each element for the two scenarios developed above. In the first instance every respondent produced answers that were distinguishable from random choices (few circular triads). This provides some reassurance that the variables presented were distinctive to those completing the questionnaire and that the group was making informed choices.  It can’t be ruled out that some respondents understood the logic of what they were doing which enabled them to minimise triads but they were not necessarily making informed choices.

Variance in answers is generally greater for the ‘livelihoods’ variable than the ‘life-saving’ one (major ellipse axis in y direction), particularly for elements that are concerned with physical scientific monitoring or knowledge. There is comparatively little difference between responses for each of the two scenarios. During the exercise there was complaint that the two scenarios were insufficiently different. So, if this exercise is repeated the second scenario should be made more different or only one scenario should be developed. While constructing the scenarios we debated making the two scenarios have different time periods within the same eruption (one month in and, say, six months in for example). This might have helped in terms of making choices about ‘livelihoods’. It is perhaps easier to frame choices around a more protracted eruption that has pulses of heightened activity.

Generally, the ability to communicate escalating risks and uncertainty to those impacted was regarded as important both in saving lives and in enabling populations to adapt and cope. The ability to have flexible infrastructure including the provision of alternative land was uniformly seen as important in enabling populations to adapt and cope with both scenarios. Rather surprisingly, the least valued element was the ability to declare the eruption over; again this might be a stronger reflection of the choice to have two different, early stage eruptions developed in the scenarios.

Differences among experts

Figures 3 and 4 illustrate differences between the expert groups across the two variables for Scenario 1 only. The mean ranking score is shown in the first figure and then the degree of agreement and concordance illustrated in the lower. The greater variance of opinion for the individual elements in Figures 1 and 2 is also reflected in the lower scores for concordance and agreement in Figure 4 (‘livelihoods’). The greater heterogeneity in views in the ‘interdisciplinary’ grouping is also reflected by their comparatively low agreement and concordance in Figure 3.

There are some striking differences between the groups, most notably between the scientific advisors and the physical scientists in ranking elements relevant to saving lives and between the interdisciplinary scientists and other groups in ranking elements relevant to allowing populations to adapt and cope with stresses imposed by activity.

The scientific advisors placed relatively high weight on being able to judge uncertainty around hazard-related information and to communicate escalating risk effectively relative to the other physical scientists.  For this variable there was relatively little coherency in the lower ranked elements, the group were clear they were ranked below the top 5 but there was less concordance or agreement about their relative position.

The communication of risk remained a dominant element for scientific advisors in helping populations to adapt and cope but for all groups the most important element was flexible infrastructure. However,  this was only just ranked more highly than public participation in decision making and good social cohesion by the interdisciplinary scientists who also saw the smallest relative drop in their degree of concordance and agreement.  In thinking about adaptation and coping the scientific advisors have a very low level of agreement or concordance between their choices, again there is broad agreement about the two most important elements but little coherence about the relative position of the lower ranked variables.

It is perhaps worth mentioning that the relative agreement across such a disparate disciplinary group overall bodes well for a good shared understandings of these problems!


Initial Discussion

One of the overall aims of the STREVA project is to come up with improved analysis of risk and a means to understand the ways in which risk changes during the course of an eruption. Ultimately, it would be good to generate both qualitative and quantitative measures of changing risk.

QRA will inevitably involve some choices about what the critical elements are to include and quantify in that analysis and to consider some means to describe uncertainty around the choice of these elements and their variance.

This paired comparison was intended as a first step in this process and in particular it was hoped to use it to analyse the relative importance of some elements and to provide a measure of variance in opinion among experts from similar and differing fields. These first results seem promising in that regard. It would be good to repeat this exercise at our next two case study volcanoes (St. Vincent and Galeras).

Another strength from this exercise is how it has highlighted our relative confusion about the difference between risk as defined by emergency actions and risk as defined by longer term coping strategies. With improved monitoring these two foci are perhaps not as disparate as we might think and this could perhaps bear more analysis. For example it would be interesting to analyse the actions and judgements around global volcanic activity in the past..

Questions to consider

  1. Does the paired comparison tell us anything meaningful?  We need to systematise our approach so that we can get some value from the comparisons between the different settings.
  2. How would this help to develop the risk analysis?
  3. Would it be more useful to use the paired comparison for actual past activity for our forensic volcanoes?
  4. Should repeat this exercise with different groups of experts with differing perspective (e.g. risk managers).
  5. In providing some focus to think about the difference and similarities between considering risk in relation to emergency actions and in coping in the longer term should we reconsider how we frame these two variables?