Crisis management training using discrete-event simulation and virtual reality techniques

Crisis management training using discrete-event simulation and virtual reality techniques

Computers & Industrial Engineering 135 (2019) 711–722 Contents lists available at ScienceDirect Computers & Industrial Engineering journal homepage:...

NAN Sizes 0 Downloads 9 Views

Computers & Industrial Engineering 135 (2019) 711–722

Contents lists available at ScienceDirect

Computers & Industrial Engineering journal homepage:

Crisis management training using discrete-event simulation and virtual reality techniques


Pak Ki Kwoka, Mian Yana, , Bill K.P. Chanb, Henry Y.K. Laub ⁎

a b

School of Intelligent Systems Science and Engineering, Institute of Physical Internet, Jinan University (Zhuhai Campus), Zhuhai 519070, China Department of Industrial and Manufacturing Systems Engineering, The University of Hong Kong, Hong Kong, China



Keywords: Collaborative simulation-based training Crisis management Discrete-event simulation Simulation application in industry Virtual reality Internet of things

Conducting emergency drills in an actual environment is workforce and resources intensive. Hence, organisations often hesitate to conduct emergency exercises frequently. Because of the limited number of opportunities to conduct drills in a year, the content of the emergency drills can only focus on common cases and exclude rare cases. This constraint also restricts the members of crisis response teams from exploring and verifying new methods for tackling a crisis. Therefore, this research uses information and communication technology (ICT), virtual reality (VR) and discrete-event simulation (DES) technologies to develop a hazard simulation system with the capability to recreate large scale and multi-agency emergency incidents that would be otherwise too costly, complex and dangerous to reproduce in the actual system. With this system, organisations can conduct emergency drills inside a virtual world having a close correspondence with their real physical apparition. This training method is called virtual collaborative simulation-based training (VCST). Two laboratory-based studies were conducted to examine user perceptions about the VCST method. A total of 60 university students majored in managerial-related subjects were enrolled, and the results showed that the proposed method appears to be a feasible approach for practising crisis management training. Further research is needed to verify the findings in larger samples and different populations.

1. Introduction Emergencies are serious, unexpected, and sometimes highly catastrophic events that require our prompt actions. Organisations often conduct safety drills to ensure their crisis response team can provide a rapid, effective, and appropriate response to the situation according to their standard emergency response procedures when an emergency occurs. The primary objective of these safety drills is to provide their staff hands-on experience on the trial emergency incident and opportunities to practise their crisis management skills in a safe environment by replicating the potential crisis in the actual system. While this kind of training is an essential step in the training and learning process referring to Miller’s pyramid (Wilford & Doyle, 2006), it always consumes a considerable amount of human and material resources to make the drill look real and genuine. Furthermore, since conducting drills using actual system often disrupts the normal operation of the system, they are often arranged at night to minimise their impact on the normal operation. As the opportunity cost of holding an emergency drill is high, organisations often hesitate to conduct emergency exercises frequently. Such limitation restricts researchers, emergency response team

members, and stakeholders from freely practising the emergency response procedures and exploring new strategies in tackling the crisis. Thus, this research makes use of information and communication technologies (ICT), virtual reality (VR) and discrete-event simulation (DES) techniques to build a hazard simulation system for organisations to conduct emergency drills inside a virtual world. The virtual world can simulate various potential emergencies inside the organisations, including those that could be too costly, complex and dangerous to reproduce in the actual system. Trainees can explore, execute, and evaluate their crisis management plans inside a safe, controllable, repeatable, and measurable environment, without affecting the actual operation. The whole training paradigm is called virtual collaborative simulation-based training (VCST) method. When compared to conducting drills in the actual environment, the proposed VCST method does not require the actual deployment of human resources or machines such as fire trucks and ambulances for setting up the environment for the emergency drills. It also does not require volunteers to play the roles of being avatars. The reason behind is that the simulation system can replicate the hazard environment, including the vehicles, avatars, and fire, in the virtual world. Moreover, as the emergency exercises do not

Corresponding author at: Jinan University (Zhuhai Campus), 206 Qianshan Road, Zhuhai 519070, China. E-mail address: [email protected] (M. Yan). Received 11 April 2018; Received in revised form 30 May 2019; Accepted 15 June 2019 Available online 17 June 2019 0360-8352/ © 2019 Elsevier Ltd. All rights reserved.

Computers & Industrial Engineering 135 (2019) 711–722

P.K. Kwok, et al.

influence the real operating system, the training time is no longer restricted at night during non-working hours. In addition, the hazard simulation system creates an opportunity for remote training via the internet, which reduces the cost of travelling to drill exercise location for trainers and staff. This remote training feature also makes the training more accessible to foreign crisis management experts and management boards in their headquarters who are usually at a considerable distance from the exercise location. Lastly, as the emergency exercises do not influence the real operating system, the training time is no longer restricted at night during non-working hours. As the proposed VCST method has fewer constraints related to the training location and time, the trainers have more flexibility in teaching and preparing staff for rare-case scenarios that have a low probability of occurrence, whereas the staff can freely explore different strategies using their creativity. It should be stressed that this paper does not suggest the proposed VCST method should replace the existing training methods, such as emergency drills. Instead, this method should be positioned as a tradeoff solution between simple classroom games and high-cost emergency drills. It aims to provide more opportunities for members of crisis response teams to practise their crisis management skills while not scarifying much the immersion, interaction, and autonomy of the training. For a better illustration of the proposed VCST method, this research developed a prototype system focusing on the metro system which allows the stakeholders of the metro system to rehearse their crisis intervention procedures in the virtual world without affecting the actual metro traffic and training schedules. The prototype hazard simulation system consists of three core parts:

regular training to their crisis response teams to equip each member with the capabilities, flexibility, and confidence to handle sudden and unexpected events (Robert & Lajtha, 2002). In general, there are three conventional training methodologies for emergency management, namely, classroom-based training, Tactical Decision Games (TDGs), and emergency drills in the real system (Ingrassia et al., 2014; Crichton & Flin, 2001; Skryabina, Reedy, Amlôt, Jaye, & Riley, 2017). 2.1.1. Classroom-based training Classroom-based training provides trainees theoretical knowledge about the procedures of accident prevention and intervention. It usually includes the concept of crew resources management (CRM). Crew resources management training aims to improve the social and cognitive skills of team members, so as to enhance the efficiency of the team. It covers concepts such as team building, briefing strategies, situation awareness, and stress management (Murphy et al., 2018). It also addresses the ways to make decisions and break the error chain in case someone makes a mistake (Helmreich, Merritt, & Wilhelm, 1999). The trainees are expected to have a clear vision of their roles during an emergency event and the way to mitigate the adverse outcome of that event. However, theoretical knowledge alone is not enough for crisis response team members to tackle crisis events effectively. Salas, Wildman, and Piccolo (2009) criticised that many management education focused on teaching theories instead of applications. They pointed out that while lectures and theories are important, many management skills can only be acquired through practice. In particular, humans often make mistakes in a high-stress situation (Cooper, 1980). Therefore, regular practice is essential for the trainees to master the crisis management skills.

(1) A DES software which controls the development of the incident; (2) User interfaces, such as VR display devices and tablets and personal computers applications, which allow users to interact with the simulation model; (3) A database which stores the parameters and key performance indicators of the virtual drills.

2.1.2. Tactical Decision Games (TDGs) In an ideal case, organisations are wise to provide practice opportunities to their crisis response teams such that the teams can exercise the skills learnt in lecture repeatedly, improve, and maintain proficiency of the skills. By regular practice, the team members from various departments can also build agreement on the way to cooperate during a crisis. However, emergency events rarely happen. Hence, the crisis response teams do not have many opportunities to gain experience from real-life. Therefore, Crichton and Flin (2001) developed a training technique called Tactical Decision Games (TDGs), which aims at enhancing people’s non-technical skills, such as communication and coordination skills, for crisis intervention. In a tactical decision game, participants receive a map and a set of cards which represent the emergency event. They are expected to work together to settle the crisis in a short and limited time. They also need to execute their strategies on the movement of workforce and materials on the map. Tactical decision games allow leaders to receive immediate feedback from participants about their illustrations and solutions to the event (Crichton & Flin, 2001). However, the simulation exercise in a TDG is usually imprecise and verbal-focus. It does not require the trainees to execute the procedures in person. Therefore, TDGs fail to tell whether the operators can execute the instructions from their leaders correctly and the potential risk of each decision. This feature of TDGs reduces the immersion, interaction, and autonomy of the training. Salas et al. (2009) pointed out that immersion is essential in simulation-based training because it can increase trainees’ interest in playing the simulation games and prompt their relevant emotion which occurs in the real situation.

This paper involves five sections beginning with Section 1 as the introduction. Section 2 reviews the existing studies about crisis management training and simulation-based training methods as well as the simulation and VR technologies. The proposed VCST paradigm is then illustrated in Section 3. Finally, a study about the user perception of the VCST method for crisis management training is presented in Section 4, ending with a concluding summary in Section 5. 2. Literature review 2.1. Crisis management training Organisations often develop a set of emergency response procedures and disaster management plans with the aim of guiding their crisis response teams to settle the crisis. A big assumption behind these plans is that every individual could follow the procedures and react correctly under stress. However, human histories tell us that this assumption is not valid (Crichton & Flin, 2001). In particular, Robert and Lajtha (2002) specified that “the key to effective crisis management lies not so much with the writing of detailed manuals (that have a low likelihood of being used, and an even lower likelihood of being useful)”. Rouse, Cannon-Bowers, and Salas (1992) also pointed out that it is impossible to write down the procedures of every contingency. Also, it is not surprising that humans would make mistakes during emergency incidents, and these mistakes could lead to more severe consequences via chain reactions. Examples of common mistakes include inadequate situation assessment, erroneous judgements, blind allegiance to the procedures, adverse reaction under stress, unclear roles resulting in tasks falling through the cracks, and miscommunication (Crichton & Flin, 2004; Rouse et al., 1992). Therefore, organisations should provide

2.1.3. Emergency drills Emergency drills are also a kind of crisis management training methods. They provide an opportunity for the crisis response teams to face the emergencies and practise crisis management skills or procedures (such as triage, evacuation or communication) in a nearly real 712

Computers & Industrial Engineering 135 (2019) 711–722

P.K. Kwok, et al.

environment by replicating emergency events in the real system (Skryabina et al., 2017). Compared to the TDGs, emergency drills require trainees to execute the emergency procedures in person, so the management and team leader can know whether the emergency procedures are valid and whether the operators can correctly execute their instructions (Gebbie, Valas, Merrill, & Morse, 2006). Also, since the emergency drills are conducted in the real system, they can increase the sense of immersion and autonomy of the trainees to the simulated situation. Therefore, organisations often conduct drills regularly to prepare their staff for handling emergencies. However, there are several drawbacks to the emergency drill method. Firstly, Salas et al. (2009) pointed out that immersion is essential in simulation-based training because it can increase trainees’ interest in playing the simulation games and can prompt their relevant emotions which are to occur under the real situation. Therefore, organisers of safety drills often deploy many human and material resources to increase the realism of the mock environment. Secondly, since conducting drills using actual system often disturbs the normal operation of the actual system, the safety drills can only be arranged at night during non-working hours to minimise the impact on the normal operation. This feature generates extra opportunity cost and limits the time for carrying out the safety drill. This feature again increases the cost of conducting the drills. Because carrying out an emergency drill is costly, it cannot be conducted very often. This shortcoming reduces the number of opportunities for crisis response teams to gain experience. Hence, these drawbacks of emergency drills bring the need of training using computer-based simulation.

safety threats, detect the knowledge gaps, and enhance the team spirit. Murphy et al. (2018) also used simulation as a platform for training trauma teams. They pointed out that trainees could transfer the skills learnt in the simulation-based training program to real-life applications, such that the time for the critical operation was dramatically reduced. Tactical decision game and emergency drills both belong to this category. However, this training method often requires a considerable amount of human and material resources to make the drill look real and genuine. With the development of computer technologies, situated or experiential learning in a virtual environment become feasible. The computer-based simulation training requires computer technologies and involves a virtual world which replicates the real system. Trainees can practise their skills inside the “sandbox” without any concern about affecting the actual environment. (“Sandbox” is a term in the field of computer security and software development. It is an isolated computing environment for testing a program securely without affecting the normal operation (Rouse, Thompson, & Tselly, 2005)). Heinrichs, Youngblood, Harter, and Dev (2008) suggested the major benefits of conducting training in a virtual environment. Firstly, the training does not require all trainees to be present at the same time and location. In particular, it does not require extra human resources to perform the avatar roles, for example, serving as a patient. Secondly, the training can be carried out at any time regardless of the operation hours of the real system. Therefore, the trainer can carry out the training whenever the trainees are available. Hence, the training frequency can be increased. Thirdly, since the training can be carried out frequently, trainers can involve a variety of scenarios and conditions in training, including some rarely occurring situations. Fourthly, the training scenarios can be repeated during a short period. Hence, the trainees can refine their skills according to their mistakes in each iteration. Lastly, the performance of trainees during the simulationbased training can be captured automatically for feedback and assessment after the event. Therefore, this training technique can reduce the workload of the trainers who are traditionally responsible for monitoring the performance of the trainees. Regarding these benefits, some researchers applied computer-based simulation training to enhance skill acquisition. For example, Smith and Trenholme (2009) applied it to educate people the fire emergency escape procedures in the building, whereas Crumpton and Harden (1997) used it to teach ergonomics concepts inside a classroom. The usage of a flight simulator for educating pilots to control an aircraft in the aviation industry (Salas, Bowers, & Rhodenizer, 1998) is also an example of computer-based simulation training. While most of these studies focused on teaching specific technical skills, it is believed that nontechnical skills, such as decision making, situation awareness, communication and coordination, teamwork, and stress management, are more critical in a crisis (Crichton & Flin, 2004). This paper proposes a VCST system to train these non-technical skills.

2.2. Simulation-based Training Simulation refers to the process of creation of an artificial or synthetic environment which imitates the operation of a real-world process. The simulation technique is often used for studying a process under situations when the actual system is not available or the cost of conducting a test in the real environment is high (de Sousa Junior, Montevechi, de Carvalho Miranda, & Campos, 2019). For instance, scholars and consultants create simulation models to verify their process designs before they implement the designs in the operations, such as (Gul & Guneri, 2015; Mouayni, Etienne, Lux, Siadat, & Dantan, 2019). Simulation-based training is the use of a simulation technique to facilitate the process of skill acquisition for the staff. It tries to substitute the training in the actual system with a guided one inside an artificial and interactive environment (Lateef, 2010). Simulation-based training can be classified into three board types, namely, role-playing simulation, physically based simulations, and computer-based simulation (Salas et al., 2009). Role-playing simulation is the simplest form of simulation as it only requires participants to show their behaviour given a scenario and roles. BaFá BaFá simulation is a kind of role-playing simulation which is popular in a conflict management class (de Jong & Warmelink, 2017). Its objective is to bring attention to the effect of stereotypes and how different cultures can create misunderstanding and eventually magnify conflict. Typical BaFá BaFá simulation training divides students into two groups of different cultures, who are then asked to interact with students from the opposite group with a different culture from theirs. Through the simulation, the students build sensitive towards cultural differences within the workplace. Compared to role-playing simulation that seldom requires the help of computer program and physical props, physically based simulation requires the aid of physical props to increase the realism of the training. There are many applications of physically-based simulation-based training in the field of medicine. For example, Wheeler, Geis, Mack, LeMaster, and Patterson (2013) applied simulation-based training techniques to educate the operators of a children’s hospital with an aim to improve the quality of care given to the children. They claimed that simulation-based training could help the operators to discover potential

2.3. Virtual reality VR is an immersive computing technology that allows people to enter and experience things inside an artificial virtual world as if it were real and also get a sense of actually being there (Bowman & McMahan, 2007). Stereo-capable monitors with desktop tracking, Cave automatic virtual environment (CAVE) system and VR head-mounted displays (HMD) are typical devices that enable the application of VR (Berg & Vance, 2017). With VR, people can explore things at any time and location once the artificial three-dimensional world is set up. This feature of VR technology makes it a useful tool for learning and practising. In fact, many researchers have also studied the feasibility of using VR for skills transfer. For example, North, North, and Coble (2015) used this technology to facilitate the learning of public speaking. He created a virtual theatre filled with avatar audiences and asked students to deliver speeches in that virtual theatre. He found that this training method 713

Computers & Industrial Engineering 135 (2019) 711–722

P.K. Kwok, et al.

helped to increase students’ satisfaction with their speech delivery and reduced their public speaking anxiety. Seymour et al. (2002) and Sugand, Akhtar, Khatri, Cobb, and Gupte (2015) also studied the possibility of using this technology for medical training. They both found that VR surgical simulators used in training improved the performance of the surgical students greatly. On top of the success in knowledge transfer, Sugand et al. (2015) pointed out that provided an extra benefit of using VR simulators in training: It would not post any risk to the patient safety because it is conducted in an isolated and controlled environment as well as the time of the training is more flexible and no longer limited to clinical hours. Concerning the above advantages, VR technology is included in the proposed crisis management training system because it can provide a safe, environmental-friendly, flexible and controlled environment for trainees to practise and execute their crisis intervention strategies.

Fig. 1. A demonstration on the Dashboard tool in FlexSim.

as the “logic controller” of the proposed crisis management training system. Firstly, FlexSim is designed for accurately simulating the process of a system. Hence, programmers of the crisis management training system do not need to design the storyboard of the game engine or write scripts from scratch for every study. The software can dynamically evaluate the human flow inside the system once the modeller has built the simulation model. Second, FlexSim is capable of studying the operation of systems in various fields. Therefore, the proposed crisis management training system is also suitable for a wide range of fields. Indeed, Raghothama and Meijer (2015) also integrated a simulation package called SUMO with VR before, but their application was instead in urban and transportation planning (also in Raghothama & Meijer, 2015).

2.4. FlexSim – a discrete-event simulation software FlexSim is one of the most commonly used DES software for creating a simulation model and performing a DES experiment (Akpan & Shanker, 2017). FlexSim Software Products Inc develops this software. It can help organisations to visualise and simulate the dynamic-flow process activities, such as the flow of products and people, inside their systems (Nordgren, 2002). FlexSim is one of the few simulation software which allows users to create and present their simulation model using preprogrammed three-dimensional objects and animations (AbuTaieh & El Sheikh, 2007). This feature saves users’ effort in programming the model, and hence they can pay more attention to the simulation concept, logic, and methods (Beaverstock, Greenwood, Lavery, Nordgren, & Warr, 2011). Thus, due to FlexSim’s user-friendliness, the software is used to build the simulation model of the proposed crisis management training system. Nordgren (2002) specified five significant steps of conducting a DES experiment using the FlexSim simulation software, namely, (1) Develop a layout, (2) Connect objects, (3) Build details of the objects, (4) Run the model, and (5) Review the output. The first step can be accomplished easily by dragging predefined 3D objects from the library to the layout window according to the layout plan of the facility. After completion of the layout of the model, the relationship between each object can be specified by linking them together via several mouse-clicks allowing a clear view of all possible ways of people flow and resources flow in the modelled system. The next step is to fill in the attributes of each object, such as cycle times, capacities, and speeds and define the logic which specifies how the people and resources flow inside the system. FlexSim allows users to program the logic using C++ or flexscript (a C++ library that is precompiled). In fact, the latest version of FlexSim includes a list of predefined logic that significantly simplifies but maintains the flexibility of programming a simulation model. Upon completing the above steps, the simulation model is ready to perform experiments. It can be run to collect the relevant data that is required. Another distinctive feature of the FlexSim simulation software is that it allows users to view the process and monitor the performance, such as the utilisation of the processor, in real-time while the model is running. For example, Fig. 1 shows a tool in the software called Dashboard which can dynamically plot the utilisation of the processor in a simple system over time. Many researchers used FlexSim to study the operation of systems in their respective fields, ranging from manufacturing and warehousing to hospital management. For instance, Pan, Shih, Wu, and Lin (2015) adopted the software to test their order picking strategy in a warehouse, while Mikulik, Cempel, Kracik, and Dbal (2014) applied it (healthcare version of software) to study the process of evacuating people from a hospital building. While FlexSim is professional in helping organisations in many fields to study and improve their operations, it was seldom applied to construct a system for crisis management training, particularly in the field of mass transport system, such as the railway system. There are two advantages of using a FlexSim simulation model

3. Proposed methodology This section proposes a paradigm for crisis management training with an aim to make this training more cost-effective, safe, and flexible. The core idea of the proposed methodology is to move the emergency drills that are originally conducted in the actual environment into a virtual world. If the safety drills can be conducted in a virtual environment using the VR and DES technology, organisers of the drills no longer need to deploy workforces, equipment, machinery, and vehicles into the training scene which otherwise can be saved for a better purpose. The reason behind is that computers can generate all necessary avatars and scenarios to make the exercise look real. In addition, the remote training feature of the simulation system reduces the costs for travelling trainers, trainees, and foreign experts to the location of the emergency drills. The proposed crisis management training system combines ICT, DES and VR technologies to recreate large-scale and multi-agency emergency incidents that would be otherwise too costly, complex and dangerous to reproduce in the actual system. Trainees can interact with the virtual world using devices such as phones, tablets, VR HMD devices, VR CAVE systems and other specialised user-interfaces. One of the benefits of the proposed training paradigm is that the simulation system can generate a dynamic environment to replicate the chaos and pressure that the trainees may encounter at the emergency incident. The VR technology also gives trainees an immersive perception of the incident as if they were there. All stakeholders, especially the personnel in the command role, are required to make prompt decisions spontaneously as their actions directly influence the outcome of the events. Since the training is conducted inside a virtual world, which is isolated from the actual system, the trainees can freely test different scenarios intervention strategies using their creativity without the worry of disrupting or damaging the actual system. The system can also automatically monitor and record the performance of each trainee allowing the trainer to provide feedback in return based on data after the drill.


Computers & Industrial Engineering 135 (2019) 711–722

P.K. Kwok, et al.

Fig. 2. Use case diagram of the prototype system.

Fig. 3. System architecture of the prototype system.


Computers & Industrial Engineering 135 (2019) 711–722

P.K. Kwok, et al.

the trainer can debrief the trainees after the training. The database also serves as the bridge between the graphical interface and the FlexSim simulation model, such that the simulation model can react to users input. Fig. 4 shows the database design of the prototype.

3.1. Design of the system This research developed a prototype system focusing on the metro system so as to provide a platform for the stakeholders of the metro system to rehearse their crisis intervention procedures in the virtual world without affecting the actual metro traffic and train schedules. However, it should be stressed that the concept of this prototype does not limit to only transportation systems but can also be applied to other areas, such as factories, airports, hospitals, and conventional and exhibition centres. Fig. 2 shows the breakdown of functions and interactions between users and the system of the prototype system. According to the use case diagram, the proposed system has six actors (types of users) and thirtyseven use cases. Each user has a unique role in the safety drill and has permission to read different information. For example, the metro station manager only has permission to receive information about his/her station, whereas the shuttle bus manager only gets information about the shuttle buses. The manager of the central control room of the metro system has permission to change the number of trains on tracks but cannot directly close the entrance of a station. The reason is because it is the respective station managers’ responsibility to change the status of their stations. Therefore, trainees are expected to develop strategies to communicate effectively. Crichton and Flin (2001) pointed out that communication and teamwork are two essential skills needed for settling a crisis effectively. Among those thirty-seven use cases, ten of them are about the operations of metro stations and another eight of them are about the operation of shuttle buses. While seven of them are about the operation of the central control room of the metro system, the remaining twelve of them are about the simulation-based training itself. Fig. 3 shows the architecture of the prototype training system, whereas Table 1 also lists the resources needed to construct this system. In general, the system consists of two servers and several computers, tablets, smartphones and VR devices. One server is used for processing the simulation model, whereas the other server hosts the database and webpage-based control panels. Computers, tablets and smartphones are used for displaying the control panels. Users can execute their crisis intervention plan into the simulation model using these devices. VR devices are used for displaying the simulated crisis environment to increase trainees’ sense of immersion and autonomy in the emergency exercise. In detail, the proposed training system involves three core parts, which are the database, the DES model, and the graphical interface (control panels and VR).

3.1.2. Discrete-event simulation model The DES model functions as the nervous system in the virtual world. It is built using the FlexSim simulation software, which is designed for studying operations inside a system. For the prototype system, the simulation model replicates a metro system consisting of eight stations and two trains. When the simulation experiment starts, the simulation model continuously evaluates the human flow inside the metro system. Fig. 5 displays the map of the metro network. Fig. 6 exhibits the outlook of the simulation model. Traditional simulation experiments, such as those in Pan et al. (2015) and Mikulik et al. (2014), aim to study a process have most of the parameters fixed. The simulation experiment usually repeats several times to get an average result. In contrast, in the simulation model of this training system, the parameters and states are not fixed. Instead, trainees can adjust them using the control panel on tablets, smartphones, computer and VR devices. For instance, they can close the entrance of a station to perform crowd control if they find that the station is crowded with people. The simulation model of the training system can instantaneously react to the actions taken by the trainees and evaluate the human flow inside the metro network. It also continuously updates the latest information about the incident to the control panels, such as the utilisation of the trains and the crowd level in each station. The virtual environment right away reflects the crowded level at each location such that trainees can instantly refine their strategies. 3.1.3. Graphical interface (control panels and VR) Wilford and Doyle (2006) noted that the realism of the simulated environment affects and improves the learning efficiency of the trainees. Salas et al. (2009) also pointed out that immersion is highly imperative in simulation-based training because it can increase trainees’ interest in playing the simulation games and prompt their relevant emotion which occurs in the real situation. Therefore, while the users have different roles in the emergency exercise, most of them can enter the virtual world using VR devices, such as VR HMD devices and VR CAVE systems, to increase their sense of immersion and involvement in the drill. The user interface of the proposed system consists of two parts: VR and control panel. With VR technology, trainees can walk around the virtual world to explore the crisis scene and make instant judgments. For example, the metro platform staff can enter the virtual platform to explore what is happening on the platform (Fig. 9). Fig. 8a–b are screenshots of the virtual environment of the prototype system. Apart from VR, managers and their subordinates often execute their decisions via computers, tablets and smartphones. Therefore, the system also involves control panels, which show the latest information about the virtual metro system and displays on computers, tablets,

3.1.1. Database Steadman et al. (2006) pointed out that one of the core features of simulation-based training is that it could show responses to action. They commented that this could improve the skill acquisition comparing to verbal feedback. The database in the proposed VCST system serves this function. It records the performance of the trainees so that Table 1 Resources needed to construct the system. Resource


FlexSim simulation software

For building the simulation model

Unity, Autodesk 3ds Max and MiddleVR

For developing the VR applications


For processing the simulation model and hosting the database and webpage-based control panels


For recording the performance of trainees

Tablets, smartphones and computers

For displaying the control panels and information about the metro system. (Trainees can control the virtual metro system using the webpage-based control panels.)

VR systems such as VR HMD devices and VR CAVE systems

For letting trainees visualise and interact the virtual metro stations, bus stops and control rooms

Wireless routers, network switches and LAN cables

For linking the computer devices


Computers & Industrial Engineering 135 (2019) 711–722

P.K. Kwok, et al.

Fig. 4. The database design of the prototype (Entity Relationship Diagram).

3.2. The way to conduct the training Before the start of the emergency exercise, every participant gathers inside a conference room. The trainer briefly introduces the agenda of the exercise, the scenario they may encounter during the emergency drill, and the way to use the computer devices. Each participant is then assigned a role in the emergency exercise and receives devices, such as tablets, which are specific to his/her position in the drill. After the briefing session, each participant goes to a specific destination and is expected to exchange information using communication devices like walkie-talkies and phones. They are also required to wear the VR headsets or enter the CAVE VR environment to immerse themselves into the virtual emergency environment. When everyone is ready, the trainer starts the emergency exercise, runs the simulation model, and triggers the events in the VR environment. Fig. 8b shows that a fire breaks out on the platform in a metro

Fig. 5. A map of the train network of the prototype system.

smartphones. For example, the staff of the central control room of the metro network can arrange more trains on the track (Fig. 7a), whereas the shuttle bus manager can deploy more buses to travel the passengers to other stations (Fig. 7b).


Computers & Industrial Engineering 135 (2019) 711–722

P.K. Kwok, et al.

Fig. 6. Simulation model of the prototype system.

station. Fig. 8a shows that all lights in the metro station are off after the explosion due to short-circuit. The platform staff on the virtual platform immediately reports the incident to the metro station manager, triggers the fire alarm, and evacuates the passengers (Fig. 9). The metro station manager in the virtual station control room checks the situation via virtual CCTV and shuts down the station using the station control function of the system. The metro station manager also updates the latest condition to the central control room. The train operation manager in virtual central control room issues the red alert and asks his/her subordinates in the VR CAVE to contact the shuttle bus department to deploy buses to move passengers between stations as shown in Fig. 10a. The subordinates also notify the traffic department of the government, the press, and the passengers. After receiving the call, the shuttle bus manager dispatches shuttle buses using the bus control function in the system to serve the passengers. The shuttle bus manager also asks his/ her subordinates to take in charge of the virtual bus stop in the VR CAVE as shown in Fig. 10b. The simulation model dynamically monitors the number of passengers at each station and bus stop. The virtual environment instantly reflects the crowded level at each location as shown in Fig. 11, such that the train operation manager at the virtual central control room and the bus manager can decide how many trains and buses should be arranged to the scene, respectively. On the other hand, the rescue team rushes into the metro station to fix the problem. The rescue team puts out the fire using the fire extinguisher and fixes the power failure issue. All these actions can be replicated using VR technologies. The emergency exercise lasts until the rescue team reports all problems are fixed and the metro station manager reopens the station. At the end of the emergency exercise, all participants go back to the conference room for debriefing. Rouse et al. (1992) noted that delivering feedbacks about the performance of trainees is essential for crisis management training. The trainees can evaluate their performance based on the key performance indicators recorded in the database and compare the results with the data in history.

4. User perceptions about the VCST intervention using the hazard simulation system 4.1. Materials and methods Two independent studies were conducted to examine user perceptions about the VCST intervention using the hazard simulation system in the contexts of a public transportation setting (Study 1) and a chemical manufacturing setting (Study 2), respectively. Procedures and findings of each of the studies are described below. 4.1.1. Outcome measures Variables examining participant perceptions about the hazard simulation system were adapted from previously validated measurement scales. Perceived ease of use and perceived usefulness, drawn from the Technology Acceptance Model (Davis, 1989), were used to examine the degree to which participants believe that using the hazard simulation system would be free of effort and useful for crisis management training, respectively. Perceived behavioural control (Taylor & Todd, 1995) and application-specific self-efficacy (Agarwal, Sambamurthy, & Stair, 2000) were used to measure the external constraints and internal capabilities that participants perceived when using the hazard simulation system. VR anxiety, adapted from (Compeau & Higgins, 1995), was used to measure participants’ negative affective reactions toward the hazard simulation system such as apprehension or fear of using VR technologies. Two other variables, attitude (Ajzen, 1991) and intention to use (Venkatesh, Morris, Davis, & Davis, 2003), were employed to examine participants’ subjective feelings and intentions toward using the system to practise crisis management training. The measurement items (summarised in Table 2) were rated on a Likert-type scale ranged from 1 to 7, where 1 refers to “very strongly disagree”, 2 refers to “strongly disagree”, 3 refers to “disagree”, 4 refers to “neutral”, 5 refers to “agree”, 6 refers to “strongly agree”, and 7 refers to “very strongly agree”. 4.1.2. Participants Participants of both studies were recruited from a local university.

Fig. 7. A wireframe of the control panel on computers and tablets. 718

Computers & Industrial Engineering 135 (2019) 711–722

P.K. Kwok, et al.

Fig. 8. A screenshot of the virtual emergency drill environment (platform of a metro station).

Inclusion criteria for enrolment were: (1) age of 18 years or above, (2) had a normal or corrected-to-normal vision, and (3) major in management science, industrial engineering or other managerial-related subjects.

4.2. Results A total of 60 university students were recruited, among them, 30 were enrolled in Study 1, and the other 30 were enrolled in Study 2. Their socio-demographic characteristics were shown in Table 3. Results from the paired samples t-test showed that there was no significant difference in age (p = 0.56), self-reported daily IT usage (p = 0.91), and self-reported daily VR usage (p = 0.24) between the two study samples. Referring to Table 4, the absolute values of skewness and kurtosis of all the variables in both studies were less than 2, indicating the normal distribution of the collected data. Descriptive statistics of the variables showed that, in both studies, the lower bounds of the average ratings of perceived usefulness and attitude were higher than five at the 95% confidence level. Overall, participants’ ratings on the tested variables in Study 1 were slightly higher than that in Study 2. Results of the paired samples t-test also showed that participants’ ratings on perceived ease of use (p = 0.001) and perceived behavioural control (p = 0.005) in Study 1 were significantly higher than their counterparts in Study 2; no significant differences were found in other variables. This result could be because participants may be more familiar with the application scenarios in Study 1 (public transportation setting) than that in Study 2 (chemical manufacturing setting), which led to favourable impacts on their perceptions about the use of the system. Results of the two studies showed that the VCST intervention using the hazard simulation system appears to be a feasible approach for practising crisis management training. As reflected by the questionnaire scales, the descriptive statistics of the variables implied that participants tended to “agree”: (1) using the hazard simulation system to practise crisis management training was useful and free of effort; (2) they had the sense of control while using the hazard simulation system to perform crisis management training; and (3) they hold positive attitudes and intentions toward using this kind of intervention for future crisis management training practices. However, participants showed relatively less belief in their internal capabilities in using the system. This result may due to the majority of the participants were not entirely familiar with the use of hazard simulation system as revealed by the sample characteristics that around two-thirds of the participants did not have prior experience with VR technologies and none of them rated

4.1.3. Study procedures Both studies received institutional review board approval and obtained the informed consents from all participants. The two studies were conducted in laboratory settings and followed the same procedures. Socio-demographic information of the participants was first collected including age, gender, and experience with information technology (IT), VR, emergency events (if any), and crisis management training (for example, fire drill). Participants were required to watch a well-designed video clip that gives a systematic introduction to the hazard simulation system and how one can use the system to practise crisis training (Fig. 9–10 were captured from the video clip.). A brief training session was then carried out to teach the participants how to set up and manipulate the hazard simulation system in the public transportation scenario (Study 1) and the chemical manufacturing scenario (Study 2), respectively. After that, participants were instructed to complete a series of crisis management training tasks using the system and answer a questionnaire comprised of the seven variables mentioned above. 4.1.4. Data analysis Data collected from both studies were carefully entered and analysed to examine participant perceptions about the VCST intervention using the hazard simulation system in different application settings. Descriptive statistics including means, standard deviations, and 95% confidence intervals of the average ratings for each of the examined variables were calculated. Skewness and kurtosis of the variables were computed to examine the normality of the data. Paired samples t-test was performed to examine the significant differences in the sample characteristics and participant perceptions between the two study samples. IBM SPSS statistics package for Window was used for data analysis.

Fig. 9. A photo showing a trainee interacting with virtual emergency drill environment using a VR HMD device (The platform assistant of the metro station triggers the fire alarm). 719

Computers & Industrial Engineering 135 (2019) 711–722

P.K. Kwok, et al.

Fig. 10. A photo showing a trainee interacting with the virtual emergency drill environment inside the VR CAVE system.

Fig. 11. The virtual environment instantly reflects the crowded level at each station according to the DES model.

their VR experience levels as competent or expert. Yet, all of the participants were university students and most of them reported to possess a considerable high level of IT experience in their daily life such as using smartphones and tablet computers, which could be an advantageous factor for the acceptance and adoption of the hazard simulation system among the end-users and, to some extent, helps to explain the disagreement on their perceptions about VR anxiety (average rating equals to 2.74 and 3.18 in Study 1 and Study 2, respectively) when using the system. However, caution should be made when it comes to

elderlies or populations with lower educational levels because more mental effort may be required when dealing with such novel technologies. In addition, sufficient and adequate training sessions to facilitate targeted end-users to familiarise themselves with VR technologies and the application scenarios right before the formal crisis management training practice could be a way to increase users’ self-efficacy.

Table 2 Variables and measurement items. Variable


Perceived ease of use (PEOU)


1: 2: 3: 4:

Perceived usefulness (PU)


Using the hazard simulation system improves your ability to react in case of crisis properly Using the hazard simulation system helps you save time in managing crisis Using the hazard simulation system enhances your effectiveness in managing crisis You find the hazard simulation system to be useful in practising crisis management

Perceived behavioural control (PBC)


Application-specific self-efficacy (ASSE)

ASSE 1: You would feel comfortable using the hazard simulation system to perform crisis management training on your own ASSE 2: If you wanted to, you could easily operate the hazard simulation system to perform crisis management training on your own ASSE 3: You would be able to use the hazard simulation system to perform crisis management training even if there was no one around to show you how to use it

VR Anxiety (VR-A)


Attitude (ATT)

ATT 1: Using the hazard simulation system is a good idea ATT 2: Using the hazard simulation system is a wise idea ATT 3: You like the idea of using the hazard simulation system

Intention to use (ITU)

ITU 1: If possible, you intend to use the hazard simulation system in the future ITU 2: If possible, you predict you would use the hazard simulation system in the future ITU 3: If possible, you plan to use the hazard simulation system in the future

1: 2: 3: 4:

1: 2: 3: 4:

Learning to use the hazard simulation system is easy for you You find it easy to get the hazard simulation system to do what you want it to do It is easy for you to become skilful at using the hazard simulation system You find the hazard simulation system easy to use

You would be able to use the hazard simulation system Using the hazard simulation system is entirely within your control You have the ability to make use of the hazard simulation system You have the knowledge necessary to use the hazard simulation system

1: 2: 3: 4:

You feel apprehensive about using VR techniques It scares you to think that you could lose a lot of information using VR techniques by hitting the wrong key You are hesitant to use VR techniques for fear of making mistakes you cannot correct Using VR techniques is somewhat intimidating to you


Computers & Industrial Engineering 135 (2019) 711–722

P.K. Kwok, et al.

(3) The environment of the emergency exercises is isolated from the actual operating system. Hence, the drill no longer disrupts normal operation and business. (4) The crisis simulation system can generate events of significant scale in the virtual world, including large-scale disasters which would be difficult to replicate in reality under the actual operating system. (5) Participants and stakeholders can flexibly evaluate different crisis intervention methods using their creativities and test hypothetical or rare to occur scenarios without worrying about damaging the actual system. (6) Since this training method does not affect the actual operation, it has fewer constraints on the time and location of the emergency exercise. (7) Due to the low opportunity cost of conducting an emergency exercise, the trainers can repeat the training during a short period and allowing the trainees to refine their skills according to their mistakes in each iteration. (8) This training method can reduce trainers’ effort to monitor the performance of the trainees during the drill as the computer can automatically collect data about the performance of each participant. The trainers can provide feedback to the trainees based on this data.

Table 3 Sample characteristics Characteristics

Study 1 (n = 30)

19.4 ± 0.9 Age, mean ± SD Gender, n (%) Male 20 (66.7) Female 10 (33.3) IT experience, n (%) Absence 0 (0.0) Presence 30 (100.0) 389.6 ± 132.0 ∗ Duration per day, mean ± SD Self-reported IT experience level, n (%) None 2 (6.7) Beginner 16 (53.3) Competent 11 (36.7) Expert 1 (3.3) VR experience, n (%) Absence 19 (63.3) Presence 11 (36.7) 7.9 ± 33 ∗ Duration per day, mean ± SD Self-reported VR experience level, n (%) None 20 (66.7) Beginner 10 (33.3) Competent 0 (0.0) Expert 0 (0.0) Experience with crisis, n (%) Absence 26 (86.7) Presence 4 (13.3) Experience with crisis management, n (%) Absence 15 (50.0) Presence 15 (50.0) ∗

Study 2 (n = 30)


19.2 ± 0.9


20 (66.7) 10 (33.3)

– –

0 (0.0) 30 (100.0) 385.0 ± 180.5∗

– – 0.91

0 (0.0) 15 (50.0) 14 (46.7) 1(3.3)

– – – –

20 (66.7) 10 (33.3) 0.7 ± 2.1∗

– – 0.24

26 (86.7) 4 (13.3) 0 (0.0) 0 (0.0)

– – – –

25 (83.3) 5 (16.7)

– –

11 (36.7) 19 (63.3)

– –

This study also brings the following contributions to the field of research in crisis management, VR and computer modelling and simulation.

• First, it proposes a VCST method to make the training more flexible. • Second, it is one of the few studies which connects ICT, DES and VR.

Note: The unit is minutes. SD indicates standard deviation.

5. Conclusion and contributions Regular training is essential to prepare the members of the crisis response teams to handle emergency events. This paper proposes using ICT, VR and DES techniques to build an interactive virtual world and a hazard simulation system to educate the members of emergency response teams the way to handle the crisis. The proposed method creates a virtual environment for the participants to settle an emergency event together. It aims to make crisis management training more flexible such that trainees can have more opportunities to practise and refine their skills and ultimately reducing the number of severe operational accidents in the future. The proposed training method should provide the following advantages:

• •

(1) Organisations can conduct emergency drills in a virtual environment, where the computer systems and related user interfaces replicate the actual environment in high detail. As a result, it requires much less workforce and materials to make the drill look real. (2) The usage of ICT enables remote training via the internet. This feature reduces the cost of travelling teachers, trainees, foreign experts, and management boards from remote locations to the training scene.

In particular, the FlexSim software is often applied to study business operations, whereas the VR technique is applied to visualise operation. In contrast, this proposed system focuses on training the staff’s ability to deal with emergencies in an incomplete information environment jointly. Third, this paper presents a prototype system for demonstrating the idea of the proposed training method. The prototype system portrays how emergency drills can be conducted using ICT, DES and VR techniques. This concept can also be applied to many areas, such as factories, airports, hospitals, conventional and exhibition centres. Forth, two studies were conducted in this research to examine the user perception of the proposed VCST method. As a result, the VCST method appears to be a feasible approach for practising crisis management training.

For further research, experiments are being conducted to obtain more data to verify the effectiveness of the training using this proposed method. As an example, a control experiment can be performed to compare the learning efficiency of staff trained by the two methods – drills in the actual system and drills in the virtual environment. Psychological studies should be designed to look into the level of stress and sense of immersion of the participants in a virtual drill. In addition,

Table 4 Descriptive statistics of the variables in Study 1 (n = 30) and Study 2 (n = 30). Variable

Mean (SD) Study 1


5.58 5.52 5.39 4.71 2.74 5.78 5.38

(0.93) (0.74) (1.06) (1.02) (1.04) (0.72) (0.72)

95% CI Study 2

4.88 5.46 4.67 4.47 3.18 5.54 5.13

(0.68) (0.73) (0.56) (0.92) (0.94) (1.03) (0.95)

Study 1 [5.24, [5.24, [5.00, [4.33, [2.35, [5.51, [5.11,

5.93] 5.79] 5.79] 5.09] 3.13] 6.05] 5.64]

Skewness Study 2 [4.62, [5.19, [4.46, [4.12, [2.83, [5.16, [4.78,

5.13] 5.73] 4.89] 4.81] 3.52] 5.93] 5.49]


Study 1

Study 2

Study 1

Study 2



−0.53 −0.53 −0.26 −0.10 −0.29 −0.71 0.13

0.76 0.30 −0.06 0.68 −0.32 −0.33 0.04

−0.16 1.52 −0.58 −0.88 −1.11 0.45 −0.01

0.59 −0.72 1.05 1.64 −0.45 −0.28 0.95

3.537 0.288 3.015 0.962 −1.662 1.082 1.110

0.001 0.775 0.005 0.344 0.107 0.288 0.276

Note: SD indicates standard deviation, CI indicates confidence interval, p-values in bold indicate significance at the 0.05 level. 721

Paired t-test

Computers & Industrial Engineering 135 (2019) 711–722

P.K. Kwok, et al.

further studies should also be done to explore alternative software which can further improve the integration of DES and VR and hence further improve the proposed hazard simulation system. Last but not least, further research can be performed to explore the feasibility of integrating the proposed hazard simulation system with decision support systems to help crisis response teams to evaluate and obtain the best crisis management approach.

(2014). Education and training initiatives for crisis management in the european union: A web-based analysis of available programs. Prehospital and Disaster Medicine, 29, 115–126. Lateef, F. (2010). Simulation-based learning: Just like the real thing. Journal of Emergencies, Trauma and Shock, 3, 348–352. 70743. Mikulik, J., Cempel, W. A., Kracik, S., & Dbal, D. (2014). A simulation model for emergency evacuation time and low-cost improvements of a hospital facility using flexsim healthcare: A case study. In J. Korbicz, & M. Kowal (Eds.). Intelligent systems in technical and medical diagnostics (pp. 333–342). Berlin, Heidelberg: Springer, Berlin Heidelberg. Mouayni, I. E., Etienne, A., Lux, A., Siadat, A., & Dantan, J.-Y. (2019). A simulation-based approach for time allowances assessment during production system design with consideration of worker’s fatigue, learning and reliability. Computers & Industrial Engineering. Murphy, M., Curtis, K., Lam, M. K., Palmer, C. S., Hsu, J., & McCloughen, A. (2018). Simulation-based multidisciplinary team training decreases time to critical operations for trauma patients. Injury, 49, 953–958. 2018.01.009. Nordgren, W. B. (2002). Flexsim: Flexsim simulation environment. In Proceedings of the 34th conference on winter simulation: Exploring new frontiers WSC ’02 (pp. 250–252). Winter Simulation Conference. North, M. M., North, S. M., & Coble, J. R. (2015). Virtual reality therapy: An effective treatment for the fear of public speaking. International Journal of Virtual Reality, 03, 1–6. Pan, J. C.-H., Shih, P.-H., Wu, M.-H., & Lin, J.-H. (2015). A storage assignment heuristic method based on genetic algorithm for a pick-and-pass warehousing system. Computers & Industrial Engineering, 81, 1–13. 010. Raghothama, J., & Meijer, S. (2015). Distributed, integrated and interactive traffic simulations. 2015 Winter Simulation Conference (WSC) (pp. 1693–1704). . https://doi. org/10.1109/WSC.2015.7408288. Raghothama, J., & Meijer, S. (2015). Gaming, urban planning and transportation design process. In S. Geertman, J. FerreiraJr., R. Goodspeed, & J. Stillwell (Eds.). Planning support systems and smart cities (pp. 297–312). Cham: Springer International Publishing. Robert, B., & Lajtha, C. (2002). A new approach to crisis management. Journal of Contingencies and Crisis Management, 10, 181–191. Rouse, W. B., Cannon-Bowers, J. A., & Salas, E. (1992). The role of mental models in team performance in complex systems. IEEE Transactions on Systems, Man, and Cybernetics, 22, 1296–1308. Rouse, M., Thompson, P., & Tselly (2005). Definition of sandbox. https://searchsecurity. accessed: 03 June 2018. Salas, E., Bowers, C. A., & Rhodenizer, L. (1998). It is not how much you have but how you use it toward a rational use of simulation to support aviation training. The International Journal of Aviation Psychology, 8, 197–208. s15327108ijap0803_2. Salas, E., Wildman, J. L., & Piccolo, R. F. (2009). Using simulation-based training to enhance management education. Academy of Management Learning & Education, 8, 559–573. Seymour, N., Gallagher, A., Roman, S., O’Brien, M., Bansal, V., Andersen, D., ... Blumgart, L. (2002). Virtual reality training improves operating room performance results of a randomized, double-blinded study. Annals of Surgery, 236, 458–464. 10.1097/00000658-200210000-00008. Skryabina, E., Reedy, G., Amlôt, R., Jaye, P., & Riley, P. (2017). What is the value of health emergency preparedness exercises? A scoping review study. International Journal of Disaster Risk Reduction, 21, 274–283. 12.010. Smith, S. P., & Trenholme, D. (2009). Rapid prototyping a virtual fire drill environment using computer game technology. Fire Safety Journal, 44, 559–569. 10.1016/j.firesaf.2008.11.004. Steadman, R. H., Coates, W. C., Huang, Y. M., Matevosian, R., Larmon, B. R., McCullough, L., & Ariel, D. (2006). Simulation-based training is superior to problem-based learning for the acquisition of critical assessment and management skills. Critical Care Medicine, 34, 151–157. Sugand, K., Akhtar, K., Khatri, C., Cobb, J., & Gupte, C. (2015). Training effect of a virtual reality haptics-enabled dynamic hip screw simulator. Acta Orthopaedica, 86, 695–701. Taylor, S., & Todd, P. A. (1995). Understanding information technology usage: A test of competing models. Information Systems Research, 6, 144–176. Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27, 425–478. Wheeler, D. S., Geis, G., Mack, E. H., LeMaster, T., & Patterson, M. D. (2013). Highreliability emergency response teams in the hospital: Improving quality and safety using in situ simulation training. BMJ Quality & Safety, 22, 507–514. 10.1136/bmjqs-2012-000931. Wilford, A., & Doyle, T. J. (2006). Integrating simulation training into the nursing curriculum. British Journal of Nursing, 15, 926–930. 2006.15.17.21907.

Declaration of Competing Interest All authors have no conflict of interest to report.. Acknowledgements The study was conducted with the support of the Fundamental Research Funds for the Central Universities (grant no. 21618317) and the SEED Funding Translation and Applied Research (grant no. 201711160007). References Abu-Taieh, E., & El Sheikh, A. (2007). Commercial simulation packages: A comparative study. International Journal of Simulation, 8, 66–76. Agarwal, R., Sambamurthy, V., & Stair, R. M. (2000). Research report: The evolving relationship between general and specific computer self-efficacy–an empirical assessment. Information Systems Research, 11, 418–430. 418.11876. Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human Decision Processes, 50, 179–211. Akpan, I. J., & Shanker, M. (2017). The confirmed realities and myths about the benefits and costs of 3d visualization and virtual reality in discrete event modeling and simulation: A descriptive meta-analysis of evidence from research and practice. Computers & Industrial Engineering, 112, 197–211. 2017.08.020. Beaverstock, M., Greenwood, A., Lavery, E., Nordgren, W., & Warr, S. (2011). Applied simulation: Modeling and analysis using FlexSim. BookBaby. Berg, L. P., & Vance, J. M. (2017). Industry use of virtual reality in product design and manufacturing: A survey. Virtual Reality, 21, 1–17. Bowman, D. A., & McMahan, R. P. (2007). Virtual reality: How much immersion is enough? Computer, 40, 36–43. Compeau, D. R., & Higgins, C. A. (1995). Computer self-efficacy: Development of a measure and initial test. MIS Quarterly, 19, 189–211. Cooper, G. (1980). Resource management on the flight deck. NASA conference publication. National Aeronautics and Space Administration, Ames Research Center. Crichton, M., & Flin, R. (2001). Training for emergency management: Tactical decision games. Journal of Hazardous Materials, 88, 255–266. Crichton, M., & Flin, R. (2004). Identifying and training non-technical skills of nuclear emergency response teams. Annals of Nuclear Energy, 31, 1317–1330. Crumpton, L. L., & Harden, E. L. (1997). Using virtual reality as a tool to enhance classroom instruction. Computers & Industrial Engineering, 33, 217–220. https://doi. org/10.1016/S0360-8352(97)00078-8. Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13, 319–340. de Jong, M., & Warmelink, H. (2017). Oasistan: An intercultural role-playing simulation game to recognize cultural dimensions. Simulation & Gaming, 48, 178–198. https:// de Sousa Junior, W. T., Montevechi, J. A. B., de Carvalho Miranda, R., & Campos, A. T. (2019). Discrete simulation-based optimization methods for industrial engineering problems: A systematic literature review. Computers & Industrial Engineering, 128, 526–540. Gebbie, K. M., Valas, J., Merrill, J., & Morse, S. (2006). Role of exercises and drills in the evaluation of public health in emergency response. Prehospital and Disaster Medicine, 21, 173–182. Gul, M., & Guneri, A. F. (2015). A comprehensive review of emergency department simulation applications for normal and disaster conditions. Computers & Industrial Engineering, 83, 327–344. Heinrichs, W. L., Youngblood, P., Harter, P. M., & Dev, P. (2008). Simulation for team training and assessment: Case studies of online training with virtual worlds. World Journal of Surgery, 32, 161–170. Helmreich, R. L., Merritt, A. C., & Wilhelm, J. A. (1999). The evolution of crew resource management training in commercial aviation. The International Journal of Aviation Psychology, 9, 19–32. Ingrassia, P. L., Foletti, M., Djalali, A., Scarone, P., Ragazzoni, L., Corte, F. D., ... Friedl, T.