Is Simulator Training Worth It?

Introduction

For decades, simulation has been a part of maritime bridge and engine room training. But as with many safety initiatives, its effect is sometimes difficult to quantify. We all know, both intuitively and empirically, that simulator training has value. It extends a trainee’s experience base in both typical and atypical scenarios. But while we all agree that simulator training is valuable, we also know that it comes at a cost. After all, simulators and the time spent on them are expensive. Is the cost worth the value derived from simulator training?

Cost vs. Benefit

One compelling argument applied to safety training in general is that the value of one life saved is greater than any cost – as long as it is affordable. Thus if we believe that simulator training has the potential to save one life, it is worth the costs associated with it and therefore no further analysis is necessary. But there are real problems with that line of reasoning. First, it does not provide us with any basis on which we can compare other safety initiatives. It may be that simulator training is indeed worthwhile, but that some other safety initiative can save more lives at a far lower cost. Unless we assess the costs and value of each we are unable to make informed decisions. Another issue is that without a cost benefit analysis, implementation decisions are sometimes more emotional than logical. After all, if it can be shown that simulator training actually saves money through a reduction in accident related costs or performance issues, then perhaps its use would be even more pervasive than it already is.

This is exactly the question addressed by a very interesting MET paper given by Professor Capt. Stephen Cross of the Maritime Institute Willem Barentsz (MIWB) in West Terschelling, The Netherlands. Prof. Capt. Cross’ paper, “Aspects of Simulation in Met – Improving Shipping Safety and Economy”, was presented at the IMLA 20 conference on maritime education and training in July, 2012 at the MIWB (where I had the good fortune to meet Capt. Cross). His paper presents a concrete view of the economic effects of simulator training. The results are compelling.

The Idea

Prof. Capt. Cross expressed the motivation of his study as follows:

“If simulator training can improve safety of operations, this would result in fewer accidents, which in turn will save funds, which could be used to afford the additional training efforts.

Additionally if the amount of the increased costs of training is compared to the funds spent presently on damages from accidents, a simple cost benefit analysis could show if such training efforts are worthwhile”.

The Study

In order to answer this deceptively simple question, Cross needed to look at a wide array of information related to the desired objectives, the current conditions of MET and maritime operations, and then had to study (and sometimes project) the consequences of change. To give you some idea as to the complexity of the study,  Cross proceeded along the following path:

1. First he determined what percentage of maritime accidents were attributable to human error.
2. Next he determined what percentage of these accidents could be attributed to training shortcomings.
3. Next he determined what percentages of competencies could be improved by simulator training.
4. Next, he had to determine by how much the above competencies could be improved through simulator training.

Multiplying the various percentages together gave an estimate of the reduction in accidents through the use of simulator training. With that information, he could then look at the cost of simulator training in order to compare it to the cost savings through a reduced number of accidents. I summarize his analysis below. Please note that in the interest of space, only a portion of Cross’ analysis can be presented. I encourage you to read the paper for full details.

Finding the Percentages

1. Human Error: Looking first at the percentage of accidents attributable to human error, Cross arrived at 80% based partly on the following:

“… the Norwegian DAMA database of accidents for the Safeco project (EU 4th FP, Safeco, 1996) whereby from 1981 to 1996 some 5400 accidents were included and some 1100 were analysed, the division of basic causes was 80% human factors and 20% technical factors.”

… Human Error: 80% of accidents

1. Lack of Sufficient Training: Looking at how training influences accidents, Cross looked at a number of studies which evaluated the causes of accidents. Among them he cites the following three:

1. “Wagenaar and Groeneweg (Wagenaar, et al, 1987) found 35% of the accidents were due to improper training and 46% due to bad habits, which could most likely be influenced by procedural training. This totals 81%.
2. Inoue (Inoue, 1996) finds 55% of accidents to be collisions and 15% groundings. That means that implicitly 70% of the reviewed accidents could have been avoided … by better trained personnel. Although possibly technical failure can also account for some of this kind of incidents.
3. For the Safeco project (EU 4th FP Safeco, 1996), 41% of the 80% human error related accident causes, indicate lack of knowledge, skills and attitude, which could be improved by training. Additionally 37% of 80% is due to lack of operational procedures. This means that up to 33% + 30% = 63% of the investigated accidents could have been influenced and possibly partly avoided through relevant competence based and procedural training.

From these three studies it seems conservatively acceptable to say that from 65% upward of the investigated casualties has relevance to (lack of) sufficient training.”

… Lack of Sufficient Training: 65% of accidents

1. Simulator Training Applicability: Not all competencies needed for safe operations can be taught and practiced in a simulator. Thus the next step was to determine what percentage of competencies were, in fact, “teachable” via simulator training. According to Prof. Capt. Cross:

“Part of the required seafarer training can be done using simulator systems. In order to estimate which part, a count can be made of the number of competences or skills per function and level, as stated in STCW Code part A where simulators are indicated. This figure in relation to the total number of competences for that function and level gives a percentage of simulator applications. …The average of the counted percentages equals 58%.”

… Simulator Training Applicability: 58% of competencies

1. Competency Improvement Through Simulator Training: Finally, Cross needed to determine the level of improvement in performance that could be achieved through simulator training. To do so, the study provided simulator training to groups of mariners, both experienced and inexperienced. It then looked comprehensively at the outcomes of exercises for these groups over the time that they were involved in the training. In the end, both groups (experienced and inexperienced) benefitted significantly from Simulator training. In the words of Prof. Capt. Cross:

“Based on the … observations a calculated average performance improvement of 45% seems acceptable to be assumed [due to simulator training]”.

… Performance Improvement Due to Simulator Training: 45%

Putting the Numbers Together

Prof. Capt. Cross took the findings above, and then multiplied them together to arrive at a conservative estimate of the accident reduction possible via simulator training. The ultimate result, 14%, is shown in his table below:

Thus far, Cross’ analysis has estimated that through the appropriate application of simulator training, 14% of maritime accidents could be avoided. What does this mean for the economics of simulator training versus the cost of accidents?

So – Is it Worth It? The Economics of Simulator Training:

Prof. Capt. Cross indicates in his paper that there are many potential cost savings available through improved operations from simulator training, even when ignoring the potential for accidents. But to look at accident costs in particular, he cited the claims history of the International Oil Pollution Compensation Fund over the period of its existence. Even though the IOPC Funds claims represent a fraction of the cost of maritime accidents worldwide, they are well documented and thus provide a reliable source of information on accident costs. The results are impressive. According to Cross:

“Over the 28 year period of [IOPCF] observations used, at least 856 million $US have been claimed for accidents which in some way have a relationship to bridge, engine room or cargo handling procedures. …[A reduction of] 14% related to the simulator training course cost would allow for at least 376946 “average” student simulator courses to be afforded. As this figure is almost similar to the global officer population it means every officer could be afforded a simulator training course from the avoided accident claim costs of the IOPC Fund relevant accidents.”

So – if the 14% accident reduction estimate is accurate, and it is applied to the relevant IOPC funded accidents, the cost saved could provide every officer in the world with a simulator training course. And since there are far more accidents (and their related costs) than are funded by the IOPC, the conclusion is that simulator training has the effect of both reducing costs and improving safety – a win-win.

Conclusion

Prof. Capt. Cross’ analysis is a compelling argument for simulation training as both a cost-saving measure and a safety improvement measure. Even if you find an argument with one or another of the numbers presented in his analysis (and he is clear that there are many factors which would influence the 14% he arrived at), one could argue that the “margin of safety” in the analysis is very large. That is, it seems unlikely that his assessment could be so far off as to make simulation training a net cost, as opposed to a net saving. And even if it were a net cost, as unlikely as that might be, we can go back to the original visceral argument: if one life is saved, the any affordable cost is one well spent.

# # #

About The Author:

Murray Goldberg is the founder and President of Marine Learning Systems (www.marinels.com), the creator of MarineLMS – the learning management system designed specifically for maritime industry training. Murray began research in eLearning in 1995 as a faculty member of Computer Science at the University of British Columbia. He went on to create WebCT, the world’s first commercially successful LMS for higher education; serving 14 million students in 80 countries. Murray has won over a dozen University, National and International awards for teaching excellence and his pioneering contributions to the field of educational technology. Now, in Marine Learning Systems, Murray is hoping to play a part in advancing the art and science of learning in the maritime industry.