Developing an Interactive Survey Game for Informing Opinions about CCS

Developing an Interactive Survey Game for Informing Opinions about CCS

Available online at Energy Procedia 37 (2013) 7428 – 7435 GHGT-11 Developing an inteeractive survey game for inform ming oppi...

217KB Sizes 8 Downloads 14 Views

Recommend Documents

No documents
Available online at

Energy Procedia 37 (2013) 7428 – 7435


Developing an inteeractive survey game for inform ming oppinions about CCS a Anne-Maree Dowda*, Peta Ashworth A , Michelle Rodrigueza, Mia Pauukovicb, b Marjolein de Best-Walddhober , Koen Straverb and Jessanne Mastopb a

Commonwealth Scientific and Industrrial Research Organisation, PO Box 883, Kenmore, QLD, 4069, Auustralia b Energy Research Centre of the Nettherlands (ECN), PO Box 56890, 1040 AW Amsterdam, the Netherlaands

Abstract The Information Choice Questionnaire (ICQ) ( was developed to overcome criticisms that theree is a lack of representation of high quality informed opiinions in studies that investigate public attitudes to carbon dioxide d capture and storage (CCS). The method was successsful in forming informed and stable opinions but the processs was long and not engaging. This research aimed to devellop and examine the most effective way to make the ICQ more m interactive and compare Australian and Dutch respoondents. Results show similarities in opinions about CC CS across both countries and identified several areas of dessign and application which would improve the end-user expeerience. 2013 The The Authors. Authors. Published r Ltd. ©©2013 PublishedbybyElsevier Elsevier Ltd. Selection and/or and/or peer-review sibility of of GHGT Selection peer-reviewunder underrespons responsibility GHGT Keywords: CCS; online knowledge creation, eduucational gaming, ICQ

1. Introduction Carbon dioxide capture and storagge (CCS) is considered a relatively new and unknownn technology, when compared with other energy techhnologies. As such, research into understanding publiic attitudes to the technology has at times been criticiized for only representing pseudo opinions rather thann high quality informed opinion [1]. To overcome thiis, researchers in the Netherlands developed an Inform mation Choice Questionnaire (ICQ) [2] which aimed to t develop high quality informed opinions about CCS. The first large ICQ process, where this research builds on, included a policy problem which w required r goal respondents to select three potential solutions to achieve a carbon dioxide (CO2) emissions reduction for the Netherlands by 2030 [1]. Resppondents were presented with seven different viable options, each with a set of consequences. The responndents were guided through all the options by not only receiving the information piecemeal, but also enggaging with the information and evaluating all the t presented *

Corresponding author: ph +61 7 3327 4468; faxx +61 7 3327 4455; email [email protected]

1876-6102 © 2013 The Authors. Published by Elsevier Ltd.

Selection and/or peer-review under responsibility of GHGT doi:10.1016/j.egypro.2013.06.685

Anne-Maree Dowd et al. / Energy Procedia 37 (2013) 7428 – 7435

consequences of the options. From these choices they were able to make a more informed opinion about their preferences for solving the policy problem. The results of the original ICQ demonstrated more informed and stable opinions, however, the process of interaction was long and there was no incentive to proactively engage with the materials except through recruitment by marketing research firms and financial incentives to complete the survey. Therefore the challenge for this current research was to investigate whether it is possible to develop a more interactive and ‘fun’ approach to solving the policy problem, while still maintaining the integrity of the ICQ. This paper presents the results from the implementation of the interactive ICQ survey game across the Netherlands and Australia, thereby providing insights into the preferred climate change mitigation options in both countries. This paper will also elucidate how well received the web based ICQ instrument was and indentify areas for improvement for online surveys. Furthermore, the ICQ survey game is evaluated in terms of its effectiveness at informing participant opinions; the stability of those opinions when compared with earlier ICQ’s; and whether there were any differences in opinions across countries. 2. Literature review Past research has made strong arguments for the public being informed on policy decisions surrounding carbon dioxide emissions reduction, the next question is: what is the best strategy to do so and how can a large audience be reached? There are several strategies possible. Frequently used methods to investigate public perceptions are focus groups, surveys and interviews [3]. Focus groups have the advantage of supporting methodical information processing and assimilation, gives participants the opportunity to ask questions to experts, and facilitates the exchange of information. The disadvantage is that single opinions can dominate discussions and provide a minority perspective; therefore representation of a population is unlikely [4]. Open ended (semi-structured) interviews provide a flexible platform on which to source data, providing opportunity to delve and probe beyond an initial response to better understand the nuances behind a specific individual’s opinions, attitudes and perceptions. Such flexibility is valuable and helps to build rich data in analysis. The approach can be limited in that it seldom provides a practical assessment that can be utilized from researcher to researcher, across disciplines and public participation techniques [5]. Surveys on the other hand usually adhere to a standardized design that is rarely altered beyond the pilot phase as doing so would risk loss of statistical integrity of the data [5]. The problem with using a simple survey for complex issues is that surveys do not encourage comparison of information, are reported to result in a lack of information and a corresponding lack of knowledge on the part of the respondent, which may in turn lead to pseudo-opinions [6][7]. To combat the problems with conventional methods of obtaining public perceptions, researchers in the Netherlands [4] created the ICQ. The ICQ has been shown to be a useful and effective tool to obtain informed public opinions on reducing carbon dioxide emissions. One of the aims of the current research was to develop a strategy to reach a larger audience for the ICQ. Previous versions were distributed amongst the general public through citizen panels, with paid respondents. While this lead to large, representative samples it failed to make available the ICQ to interested members of the wider public. In order to achieve wider dissemination, a decision was made to develop an online version. The intent was to develop a useful tool for acquiring information for opinion forming on specific topics via a self-administered, open and interactive, and user friendly interface such as the Internet. To stimulate and engage participants the ICQ needed to incorporate visual appeal, be user friendly and provide motivational aspects in order to retain



Anne-Maree Dowd et al. / Energy Procedia 37 (2013) 7428 – 7435

interest and desire for continued exploration of the information provided. To better understand how to achieve this, a literature review was conducted focusing on educational gaming and online learning. Schaller and Allison-Bunnell [8] found that adults have a preference for information based educational games and straight forward cognitive information. Adults bring an intrinsic motivation to the learning experience and therefore want to learn in the most direct way, without too many distractions. According to several researchers [9][10][11] the key foundational aspects to successful educational gaming are: achieving goals; self-assessment; ability to compare to others; and providing rewards. These aspects became critical in our effort to develop an online interactive tool to help inform public opinion about CCS. 3. Methodology The aim of this research was to build on the earlier Dutch ICQ from 2007 and develop an interactive survey game using generally the same information as well as the most important parts of the ICQ methodology. We were keen to explore whether we could successfully motivate individuals to complete the survey game because of its high quality interactive display. At the same time we developed a similar but specific policy problem for Australia following the ICQ method from 2007. To ensure objectivity of the final options for Australia, a group of experts was convened to decide on the most appropriate seven options to successfully reduce annual carbon dioxide emissions by 50 million tons by 2030. Once the options were chosen, experts were invited to develop the consequences for one particular package and then these options were peer reviewed by a different group of experts and translated into easy to understand layman’s terms. These packages were then entered into an Australian version of the interactive game (see Fig 1).

Fig. 1. (a) Consequences for climate change; (b) Seven policy options

Although extensive analysis was conducted on both the Dutch and Australian ICQ data, only a brief summary of those findings will be presented here. The paper will focus more on the qualitative component of the research which investigated respondents’ interaction and feedback on the online survey. In order to capture the data for this part of the research, a set of semi-structured interviews were conducted through an “out loud” test with teachers, high school students, policy makers and citizens interested in climate change and energy in the Netherlands. Interview questions focused on the educational, policy and technical aspects of the tool. Content analysis was used to analyze the responses from the interview data.

Anne-Maree Dowd et al. / Energy Procedia 37 (2013) 7428 – 7435

4. Findings and Discussion 4.1 Comparing Dutch and Australian ICQs: Summary of results The results were similar to those found in the original Dutch study, as both the Dutch (n=104) and Australian (n=359) respondents show a preference for energy efficiency options and renewables, and oppose nuclear energy. Although the consequences of the different options for each country were quite different, the overall evaluation grades were similar in relative terms. When participants were asked to provide a report mark for each option (1=not favorable to 10=very favorable, with a 5.5 cut off point for a sufficient grade), the efficiency options, renewable sources and CCS score sufficient grades, while nuclear, coal to gas and the international trading scheme score insufficiently. These findings also reflect similar results from other international research which investigated the preferences for energy technologies to address climate change [12]. Concerning CCS it is apparent that this was not the preferred option by the public when it comes to mitigating CO2 emissions. However a difference exists between the Dutch and Australian sample. The Dutch respondents graded CCS with coal among their least preferred options, whereas the Australian respondents ranked CCS as an average option, scoring above “Nuclear energy”, “replacing future planned coal-fired power stations with gas” and “participating in an international emissions trading scheme”. A possible reason for this difference in evaluation could be that the Australian CCS option contained 10 consequences of which half were evaluated positively, compared to only one positively evaluated consequence out of six in the Dutch ICQ. This difference in information density might be the reason for the dissimilar CCS evaluations. An alternative explanation for the differences in opinion towards CCS between Australian and Dutch respondents could be associated with the conclusions drawn in a recent study on knowledge and perceptions of CO2 and CCS acceptance [13]. This study found greater awareness of CCS in the Netherlands which was explained by the mass national coverage of a CCS project in the town of Barendrecht. In this case, public protests were held about the project which eventuated in the project being cancelled. Therefore, a more negative result in this research could also be related to an overall negative reaction to the Barendrecht situation. This interpretation is also reflected if one compares only the CCS evaluation of the 2007 Dutch ICQ with the Australian ICQ. In this comparison, the differences are fewer because Dutch respondents evaluated CCS higher in 2007. These results suggest a possible negative trend towards CCS, or at least onshore CO2 storage, from Dutch respondents. Keep in mind though that the 2012 sample was small, and the difference might reflect the smaller second sample size instead of a trend, or might reflect the difference in the CCS options and consequences. 4.2 Feedback on the information provided and ICQ method At the end of the ICQ respondents were asked to give their feedback on the survey. Six questions were asked concerning the information in the ICQ (Table 1) and four questions concerning the method of the survey and use of the tool (Table 2). Questions were asked on a 7-point scale (1=negative to 7=positive). On average, all respondents indicated that they felt they had sufficient information to make a choice between the options. Respondents felt the information received was mainly impartial, clear and the text primarily met their information needs on the topic. When asked whether the amount of information provided was appropriate, the average rating was around the scale midpoint which indicated that people felt they had received the right amount of information. Respondents were also slightly more positive when asked whether the information tool had helped them to form an opinion on the topic.



Anne-Maree Dowd et al. / Energy Procedia 37 (2013) 7428 – 7435

Table 1. Feedback on information provided Survey Questions


The Netherlands





Sufficient information to make a choice between the options





Impartial information





Clear information





Information needs on the topic





Appropriate amount of information





Information was helpful in forming an opinion





In order to provide feedback on the ICQ method, respondents were asked to provide evaluation on four key areas: simplicity, pleasantness, recommend to others and learning. To the question on whether respondents found the method simple or complicated, respondents found it slightly complicated, mainly associated with the technical interface between the website and survey. Yet respondents also thought that the survey was reasonably pleasant to use and were likely to recommend the tool to a friend or colleague and likely they would use a tool like the ICQ again to learn about a topic. Table 2. Feedback on the ICQ method Survey Questions


The Netherlands















Recommend to a friend or colleague





Use the ICQ again to learn





4.3 Effectiveness of the ICQ online tool As the ICQ addresses a complex topic with high information density, the educational potential of this tool was considered. Due to the length, topic, and previously tested language comprehensibility of the tool, it was considered to be well suited for a range of educational organizations. The tool was also considered to be of use to inform policy makers and citizens interested in climate change and energy policy and production. From an educational tool perspective various scholars agreed that the grading of the different policy options and the process of choosing three preferred options following this has merit. Not only did students work through the ICQ consequences of the options quite easily, they also had discussions about it as they progressed; communicating their opinion on the given information and ‘provoking’ reactions from their fellow students to raise discussion on the topic. This position was also supported by teachers noting that it would be a good opportunity for engaging students in learning about a topic but they also indicated that sufficient classroom time would be needed for this kind of activity. One teacher stated: “These days you need to stimulate students by practical and interactive content, and this tool would be a useful way of educating.” During explanations about the background of the ICQ and the way information was generated, several teachers stated this was important to them. Because of the careful and balanced construction of the

Anne-Maree Dowd et al. / Energy Procedia 37 (2013) 7428 – 7435

content of the ICQ they were more willing to try the tool, and perceived it as responsible lesson material. One particular teacher explicitly requested information about all the organizations and experts involved in constructing the ICQ, as he wanted to be assured that there was no possibility of biased information being provided to students. Fifteen students indicated that they thought some of the consequences had a somewhat ‘dull’ character. This lead to students perceiving the information as ‘meaningless’ due to the ‘overbalanced’ or ‘boring’ information of some consequences. In addition, while both students and teachers indicated that they thought the length of the ICQ (approximately an hour to complete) was adequate when used as lesson material; students stated that they thought it too long for voluntary participation (at home). Contact was made with several layers of government to determine interest in the website as a tool for informing them, their colleagues, or citizens concerning the topic. Governmental employees of the Province of Utrecht, the municipality of Utrecht and city of Utrecht were approached. As project team members explained the concept of the online ICQ and showed policy makers some of the consequences for rating, it became clear they found the information too general for disseminating to citizens; or, did not believe it to be their role to make such information accessible to the public. One interviewed policymaker of the city of Utrecht expressed an interest in the ICQ being purpose designed for citizens of his city, with consequences identified situated near the city of Utrecht. A policy maker working at the province department stated that she believed it was the role of the national government to inform citizens with this kind of information, and not that of a province. She advised the team to approach non-governmental organizations that have the common goal of informing the public concerning such a topic. A common theme among the policy makers was that they are interested in local practical problems and answers, and in communicating to citizens what they might do as individuals to improve their local living environment; and what role specific policy makers may have to assist citizens in achieving this, in this case the province, municipality or city. The online ICQ was generally considered to be too “broad [a] topic” by policy makers both for use as a tool for informing citizens, and for their own work. As a municipality employee stated: “It shows me possible consequences, but not what practical advice as a policy maker I can give, or look at. And as a policy maker I would — for example — like to have a more detailed idea on how we can stimulate local sustainable mobility, not a selection of common consequences for reducing CO2 emissions.” Policy makers also indicated they would be interested in information on economic incentives which may help stimulate regional or local sustainability or information on how certain emission reduction options could be implemented for economic advantage. There is a dispersed network of governmental and semi-governmental organizations which work on energy related issues across The Netherlands at various scales (i.e. local, provincial and national levels). These organizations do tend to have areas of specialization but sometimes work together to address energy matters. Attempting to contact certain local authorities proved to be a time consuming activity. For example, attempts were made to contact several local community governmental organizations that share a common website for information on local energy efficiency and environmental issues. After several telephone calls and emails, communication was received from one such organization requesting the provision of additional information about the online ICQ so that it could make contact to discuss further. Follow up contact on the part of the project team failed to achieve the promised feedback. In addition to all our attempts to contact various agencies, a policymaker at a provincial department advised the team to contact appropriate policy makers in the national government for dissemination of the online tool, as many local and provincial levels of government source information from the national level. It was believed therefore that a more efficient way to disseminate the ICQ was to use a top down method. That is, the use of large networks with broad and multiple governmental levels, such as the national government, as a disperser of the tool rather than trying to secure local authority participation.



Anne-Maree Dowd et al. / Energy Procedia 37 (2013) 7428 – 7435

Contact was made with several large, well-known non-governmental organizations. These organizations indicated they did not believe the online ICQ provided the practical information the organization supported, with one organizational representative noting that: “we miss the consumer perspective of what an individual can change in their own daily life to be more sustainable.” It is our belief that the probability of people spending more than 20 minutes on the ICQ is unlikely, even when an expressed interest in the topic exists. Online ‘consumers’ prefer short snippets of information that require no more than 20 minutes to process. After this period of time most people tend to lose interest. Several Dutch websites were identified that provided information on energy, however when contacted none of the websites returned emails or responded to telephone messages, nor showed any interest in the online ICQ. 4.4. Key themes for future development Four key themes emerged from the content analysis which identified areas for future development – time, variation in versions, technical issues and attractiveness of the survey. Length of current ICQ too long for most. Although people may be interested in the topic of climate change and CO2 mitigation actions, respondents often indicated a belief that the length of the online ICQ was too long to complete voluntarily. It is recommended that future uses of the ICQ format be modified considerably and aim for a 20 minute survey design. Different ICQs for different target groups. It was noted that the tool could achieve greater audience reach if it were to incorporate three options of choice, for example, ‘light’, ‘medium’, and ‘expert’, to satisfy the needs of the different interest groups. Though a valid point, such differentiation in content would require extensive time commitment on the part of experts to develop the different packages to the exacting quality required for the ICQ process, which in turn would incur extensive costs. Avoiding technical difficulties and restraints. In the current study, different limitations were experienced which prevented desired design elements from being realized as the project progressed. Therefore, it is recommended that future research teams have access to an IT specialist in all stages of the project in order to achieve an easily accessible website with embedded survey response capturing features. The combination of gathering data and making the website attractive is difficult. The concept of informing people at the same time as gathering data proved to be quite challenging. Although the project team retained all the required information in written format, which is imperative for the ICQ method, it is recommended that some components be provided in short explanatory movies or film clips. This could significantly speed up the process while introducing an enjoyable element to the tool. In addition to the copious amounts of written information provided, respondents also felt the survey process was uninspiring. It is recommended therefore to integrate the different survey components into the one website platform incorporating an engaging presentation format that is repeated across all components such that respondent interest is maintained which may result in an increase in the completion of the tool. 5. Conclusion In conclusion, much learning was achieved through this research which provided insights into Australian and Dutch opinions on energy options and the delivery of policy information via an online tool. CCS was found to be more accepted when combined with other energy efficiency and renewable solutions.

Anne-Maree Dowd et al. / Energy Procedia 37 (2013) 7428 – 7435

Respondents appeared to be positive about the information that was provided yet they also detailed several design flaws in the online design that could be improved in future developments. The most important feedback received was the use of a single website to integrate all ICQ components including preliminary and ongoing instructive information and the different survey elements; rather than delivery via separate website interfaces. This should result in a more stimulating and inspiring process for providing information to respondents and also employing additional IT elements, for example, the use of images, figures or video’s. Finally, in order to engage and hold the attention and interest of potential ICQ participants, it is essential that consideration be given to making the process shorter and pitched to different levels of understanding and interest that meet the needs of different target groups. Acknowledgements This research was part of a broad international public engagement research program on CCS, led by the Commonwealth Scientific and Industrial Research Organization (CSIRO) and funded by the Global CCS Institute. The researchers would like to thank all the Australian and Dutch participants for their involvement. The authors would also like to dedicate this paper to the late Dr Dancker Daamen, Assistant Professor at Universiteit Leiden. His contribution to the social science field was significant and he will be sorely missed. References [1] de Best-Waldhober, M. and Daamen, D. (2006). Public perceptions of preferences regarding large scale implementation of six CO2 capture and storage technologies: Well-informed and well-considered opinions versus uninformed pseudo-opinions of the Dutch public. Prepared for the NWO/SenterNovem Project: “Transition to sustainable use of fossil fuel” March 2006 [2] Neijens, P., De Ridder, J.A. & Saris, W.E. (1992). An instrument for collecting informed opinions. Quality and Quantity, 26(3), 245-258. [3] Vaughan, G.M. & Hogg, M.A. (2002). Introduction to Social Psychology (Third Edition). Frenchs Forest, Australia: Pearson Education Australia. [4] Daamen, D. (2010). Focus Groups or Information Choice Questionnaires: Which method results in better opinions? Presentation at the “How to communicate CCS?” Workshop, “Scrutinizing the impact of CCS communication on the general and local public”, Amsterdam, 20 May, 2010. [5] Halvorsene, K.E. (2001). Assessing Public Participation Techniques for Comfort, Convenience, Satisfaction and Deliberation. Environmental Management, 28(2),179–186. [6] de Best-Waldhober, M., D., Daamen, D., Ramirez Ramirez, A., Faaij, A., Hendriks, C. & de Visser, E. (2008). Informed public opinions on CCS in comparison to other mitigation options. Energy Procedia, 1(1), 4795-4802. [7] Neuman, W.R. (1986). The Paradox of Mass Politics: Knowledge and Opinion in the American Electrate. Harvard University Press, London: England [8] Schaller, D. T., Allison-Bunnell, S., Borun, M. & Chamnbers, M.B. (2002). How Do You Like To Learn? Comparing User Preferences and Visit Length of Educational Web Sites. Retrieved from [9] Clark, D., Nelson, B., Sengupta, P., & D’Angelo, C., (2009). Rethinking science learning through digital games and simulations: Genres, examples, and evidence. Presented at the Learning science: Computer games, simulations, and education workshop sponsored by the National Academy of Sciences, Washington, D.C. [10] Jovanovic, M., Starcevic, D., Minovic, M. & Stavljanin, V.(2011). Motivation and Multimodal Interaction in ModelDriven Educational Game Design. Systems, Man and Cybernetics, 41(4), 817 – 824. [11] Papastergiou, M. (2009). Digital Game-Based Learning in high school Computer Science education: Impact on educational effectiveness and student motivation, Computers & Education, 52(1), 1-12. [12] Hobman, E.V., Ashworth, P., Graham, P., & Hayward, J. (2012) The Australian Public’s Preferences for Energy Sources and Related Technologies. CSIRO: Australia. [13] Itaoka, K., Saito, A., Paukovic, M., de Best-Waldhober, M., Dowd, A-M., Jeanneret, T., Ashworth, P. & James, M. (2012). Understanding how individuals perceive carbon dioxide: Implications for acceptance of carbon dioxide capture and storage. CSIRO Report EP 118160, Australia.