Better judgment techniques

The direct estimation approach explicitly relies on judgments by line managers under the explicit facilitation of independent risk experts. This is unlike most of the other methods of quantifying operational risk which tend to rely on implicit high level assumptions and judgments which are often not considered or questioned.

The direct estimation approach encourages the use of improved judgment by allowing the obvious traps of subjective judgments to be analysed and addressed.

These traps have been well described in the literature. One good place to start is the collection of articles in Wright, George & Goodwin, Paul (eds) (1998) Forecasting with Judgment John Wiley & Sons, Chichester England.

Projecting extreme risk scenarios (and then quantifying their impacts) needs a carefully constructed mind set different from normal for the majority of business managers. For example, a line manager focussed solely on sales growth and maintaining margins is unlikely to be an effective source for extreme loss scenarios without careful explanation and appropriate scene-setting. Some will even consider the exercise of considering adverse possibilities "hypothetical", a "waste of time", or at least a distraction from what they should be focussing on.

Tip 1 - Motivation: Ensure that the business line managers clearly understand that their business head wants a serious consideration of possible extreme loss scenarios with the process outcomes and recommendations reported through to him or her.

The structured consideration of possible extreme losses may be a new experience for many line managers. Clear framing of the activities' goals can avoid unnecessary and demotivating time-wasting. The goals should include both better risk management as well as assisting risk quantification. How can early warnings of such extreme loss outcomes be detected and acted on?

Advising and adhering to a clear time-frame for the risk identification and scenario analysis is also valuable given their open-ended nature.

Tip 2 - Clear framing: Ensure the independent risk facilitators clearly frame the activities' goals (and provide an appropriate time-frame) and that these goals include better risk management as well as risk quantification.

Avoiding common judgment traps is achieved primarily by being aware of them. Commonly people predict the future in terms of immediate past experience - the so-called "anchor and adjust" heuristic. Line managers need to be reminded not to see possible adverse outcomes solely as adjustments to the latest outcome (which may be a favourable or unfavourable one) but to step back one stage further and to see the latest outcome as also just one of many.

Tip 3 - Avoid anchoring: Ensure line managers avoid limiting scenarios to those anchored to the most recent outcome where this is not fully reflective of the possible outcomes.

The literature also suggests that people may tend to be over-optimistic in forecasting. The effect of this on projecting extreme loss scenarios is not completely clear, but there appears clear advantage in informing the projections with analysis of similar third-party loss events. This can also encourage a more objective, less emotionally involved, analysis.

Tip 4 - External losses/near-losses: Use analysis of emotionally unconnected external losses and near-losses inform a more "objective" risk scenario development process.

Tip 5 - Outsider views: Avoid defensiveness and insularity by line managers by asking them to take on an "outsider" role, such as appraising the business as one of a number of possible acquisitions.

A well-noted judgment trap is conjunction error - for example, people tend to rate the proportion of the population represented by men having heart attacks as lower than the proportion represented by older men having heart attacks. The best remedy for this is a structured step-by-step approach (for example by seeking the judgment of the population proportion represented by men, then by men having heart attacks, and finally by older men having heart attacks).

Tip 6 - Divide & conquer: Break up big subjective judgments into smaller ones that can be made separately and that can capture logical connections.

In many cases there may be more than one line manager with expertise in the risk and the way it could impact the organisation. In this case judgment biases can be reduced by separately seeking the individual judgments of the expert group. There are a number of different possible techniques.

One that can be undertaken easily as part of a brainstorming session is an iterative record-discuss-revise cycle. Here participants are first asked to record their response to the framed question/goal, then sequentially to present on an aspect of their initial thoughts with this followed by group discussion, and finally to revise their initial response.

A more structured approach, known as the Delphi technique, utilises anonymity, iteration, controlled feedback, and statistical aggregation of responses. Although developed in the 1950s it is even more useful now that electronic communications simplifies distribution and aggregration of surveys/questions.

Tip 7 - Collaborative confirmation: Use multiple risk experts wherever possible both as an internal discipline and also to capture the synergy between their thinking. Use structured approaches to minimise dominance by an individual or "group-think".

Improving the risk judgment quality of the line managers is one of the key roles of the independent risk facilitators. This is both critical for risk management and risk measurement purposes.

There may well be times when even with the best endeavours the line manager judgments appear to these independent risk faciliators to be awry. The only sensible solution in this case is for the contention to be escalated to the line managers' business head. This maintains the ownership by the business division - critical to their acceptance of responsibility for the risk - and yet also defends the risk quantification process.

To Controls & insurance

Last updated:16/5/07

Jump to... Top | Risk Management Home | Austega Home