From action icon to knowledge icon: Objective-oriented icon taxonomy in computer science

From action icon to knowledge icon: Objective-oriented icon taxonomy in computer science

Displays 39 (2015) 68–79 Contents lists available at ScienceDirect Displays journal homepage: From action icon to kn...

1MB Sizes 0 Downloads 84 Views

Displays 39 (2015) 68–79

Contents lists available at ScienceDirect

Displays journal homepage:

From action icon to knowledge icon: Objective-oriented icon taxonomy in computer science Xiaoyue Ma a,⇑, Nada Matta b, Jean-Pierre Cahier b, Chunxiu Qin a, Yanjie Cheng a a b

School of Economics and Management, Xidian University, 266 Xinglong Section of Xifeng Road, 710126 Xi’an, Shaanxi, China UTT (Université de Technologie de Troyes) ICD/Tech-CICO Lab, BP 2060, 10010 Troyes, France

a r t i c l e

i n f o

Article history: Received 4 February 2015 Received in revised form 4 August 2015 Accepted 29 August 2015 Available online 29 August 2015 Keywords: Icon Icon taxonomy Action icon Knowledge icon Graphical interface Human–computer interaction

a b s t r a c t Icon plays a critical role in computer interface design. Studies on icon taxonomy explain the way in which various types of icon represent the objects and provide designers creation rules by which icons are more in line with users’ cognitive psychology. However, along with larger and larger use of icons, the previous classification criterion causes the boundary between categories blur. What’s more, Single classification standard is not able to well illustrate the icons applied in today’s computer applications. The purpose of this paper is to present an objective-oriented icon taxonomy which proposes to categorize icons into action icon and knowledge icon. To assess this proposition, we analyzed a sample of icons that applied in computer interface and suggest precise application domains to both action icon and knowledge icon categories. The results of this practice manifested that action icon and knowledge icon implied a high relation with applied environment and explicated the development trace of computer icons. This work is one of the first to point out the notion of knowledge icon and to highlight the importance of objective of icon application. Findings in this paper could enrich icon use in computer interface design, especially provides possible way to improve online knowledge sharing by visual tool like icon. Ó 2015 Elsevier B.V. All rights reserved.

1. Introduction The principle of iconic representation may be partly demonstrated from the evidence of imaged cognition [1]. ‘‘People often recognize pictures of things and understand them more quickly than they do verbal representations of the same things.” [2] Early research in Dual-coding theory [3] postulated that both visual and verbal codes for representing information are used to organize incoming information into knowledge that can be acted upon, stored, and retrieved for subsequent use. The theory showed that memorization for some verbal information is enhanced if a relevant visual is also presented or if the learner can imagine a visual image to go with the verbal information. Likewise visual information can often be enhanced when paired with relevant verbal information. Several definitions offered by experts tried to make clear boundaries between the terms of icon and symbol [4,5]. For example, Horton considers icons as a subset of symbols [6]. McDougall argues for an inverse subordination between these two concepts: ‘‘For the sake of simplicity, icon is the term used [. . .] to refer to the broad range of icons, signs, or symbols used to help individuals ⇑ Corresponding author. E-mail address: [email protected] (X. Ma). 0141-9382/Ó 2015 Elsevier B.V. All rights reserved.

interact with machines and their environment” [7]. Marcus argues for a distinction between an icon and a symbol in terms of the concreteness of the representation: ‘‘Icons are signs that are familiar, are easy to understand, and are often concrete representations of objects or people. Symbols are signs that are often more abstract and require specific instruction to learn” [8]. In this paper, we follow Horton’s definition of icons to consider ‘‘icon” is a general reference of visual symbol. Particularly, we define that the symbolic characters of icon imply how an icon signifies the object while the graphical characters referring to the graphical variable used in an icon, like color and shape. Computer icon (henceforth ‘‘icon”) plays a critical role in Graphical User Interface (GUI) [9]. It is a group of icons displayed on the computer screen in conjunction with computer windows, menus and a pointing device form of computer system and enables the user to easily and intuitively navigate the system. One of the most notable icon designers, Susan Kare was quoted saying ‘‘good icons should be more like road signs than illustrations, easily comprehensible, and not cluttered with extraneous detail” [10]. Studies on icon taxonomy provided deeper theoretical explanation of iconic representation from the view of its characteristic. They illustrated what kind of icon existed and how each kind of icon signified the target through symbolic characters and graphical characters. The findings in turn served later icon design and

X. Ma et al. / Displays 39 (2015) 68–79

implied why some icons were better accepted than others. However the previous studies focused more on physical appearance of icons themselves instead of considering the influence on applied environment. Moreover, the former findings were carried out based on simpler computer background where user’s perception was single and icon needs was direct. Following the diversity of users and icon applications, the static icon-character-oriented taxonomy could not satisfy completely icon research. Besides, the development of visualization and knowledge engineering creates more changes for icons. They are employed not only for user’s operation guide but also in use of knowledge representation, in order to enhance knowledge understanding and knowledge reuse under sharing environment. This is one of the reasons to form two icon sets which have a strong relationship with icon applying trends: action-oriented and knowledge-oriented. Consequently, we are proposing an objective-oriented icon taxonomy, which highlights the use purpose of computer icons rather than their graphical characters or symbolic characters. This applied-domain-focused icon taxonomy is supposed to improve computer icon serving for information visualization. On one hand, new categorizing criterion will enrich icon taxonomy study. How one icon could represent an object is not the only analysis point of computer icon any more. Where this icon is able to be applied and why it occurs on this kind of interface design is also interesting to be emphasized. On the other hand, deeper illustration on icons that are suitable in each applied field will enhance the understanding on the potential of icon and in turn explore more possible iconbased interface design. The icon design could start from applying needs instead of seeking appropriate platform for the icons that have been created. This study is assumed to be meaningful for icon-based human computer interface both at theoretical level and practical level. This paper will firstly review the development path of icon taxonomy, respectively from physical appearance, user perception and representation strategy, these three main criterions. In Section 3, we will explain the proposed icon taxonomy and present two icon categories produced by it: action icon and knowledge icon. Then in the next section, a test carried out to demonstrate new icon categorizing criterion will be presented and typical application domains of each icon category will be precisely illustrated. Discussion on this objective-oriented icon taxonomy will be also analyzed in details. Finally, we conclude. 2. Background – previous studies on the taxonomy of computer icons The studies on computer icon taxonomy started in line with GUI. It was pointed out in the purpose to clarify the characters of kinds of icons applied in computer interface and tried to find the common attributes among these icons in varied appearance. The common attributes were supposed to support the creation of icon-based GUI in a diversified way. The icon taxonomy has been explored and improved for about thirty years and it formed progressively into three branches based on three different but related criterions. Under each branch, a group of icon taxonomy was proposed by different researchers. Although the findings in one group came out from various theory foundations, they built up a complete branch of taxonomy because of the overlap originating from common criterion. In this section, the computer icon taxonomy will be illustrated in the view of three criterions to present the evolution stage in the field. 2.1. Icon taxonomy based on physical appearance The studies on icon taxonomy emerged between 1980s and early 2000s. Physical-appearance-based criterion has been widely


accepted and applied for icon taxonomy since a long period. Here the physical appearance of icons refers to the relevance between iconic representation and represented object. Physical appearance of icon was the first proposed criterion in icons taxonomy and derived tens of related classification methods. Although the icon category titles of these methods differed from each other, they all implied the relevance between iconic representation and represented object as criterion. Tracing back to Lodding’s [11] icon classification, three categories were proposed: representational, abstract and arbitrary. Representational icons were defined as examples of general representing object. For example, an image of petrol pump represents a petrol pump. And abstract icons are those express the concepts rather than to display the object itself: an image of a broken glass to represent ‘‘fragile”. The third category of icon naming arbitrary was explained as the icons created and designed under certain convention. In 1986, two papers concerning icon classification were published: Gittins and Gaver. Gittins [12] analyzed icon classification based on form, type and color, but mainly focus on the form of icon. Two categories of icons were suggested by Gittins: associative icons and key icons. Associative icons were defined by the icons not only allowing computer users to identify the represented object but also to infer their graphical attributes. Gittins gave an example of this kind of icons using an image of mail trays and arrows to indicate incoming and outgoing email. Further subcategories of associative icons were also mentioned by literal icons and abstract icons. However the author did not explain them in details. Another category, key icons were those provide cognitive implication from representing objects. They were as well divided into mnemonic icons and arbitrary icons. The mnemonic icons were able to be inferred by sub-text, for example a guillotine to represent ‘‘execute”; while arbitrary icons could not be inferred. Another paper produced in this year by Gaver [13]. Gaver proposed three categories for computer icons: nomic, symbolic and metaphorical. He defined that the nomic icons have a photographic relationship with represented objects, which is similar to the representational icons of Lodding’s. And symbolic icons have an arbitrary relationship with representing object that requires be learnt and understood. Finally he illustrated that metaphorical icons use a feature of icon to represent a whole thing, such as using knife and fork to represent a restaurant. One year later, Lindgaard et al. [14] classified icons according to a simple criterion: whether abstract or depictive. Depictive icons resemble exactly the representing objects, named purely pictographic; while abstract icons is on the contrary, entitled purely symbolic. Besides these, they created a third category, mixed icons, referring to the icons have both abstract and depictive elements. Three important studies on icon taxonomy occurred in 1989 in order to present a new classification method of computer icons but somehow related to former ones [15–17]. Rogers proposed an icon classification paying emphasis on the function and form of icons. She also discussed theoretical issues of how computer users utilize information in icon-based interface displays when performing a task. Rogers in her research identified four icon types: resemblance, exemplar, symbolic and arbitrary. Rogers described resemblance icons as icons that present their underlying referent using an analogous image; she gave the International road sign for falling rocks as an example. Rogers defined her second icon type, exemplar, as icons that show only the most central attributes of an object, such as a knife and fork for a restaurant sign. Rogers defined her third icon type, symbolic, as icons whose function is to ‘convey an underlying referent that is at a higher level of abstraction than the image itself’. The image of a broken wine glass to imply fragility was offered as an illustration of this icon type. Finally Rogers


X. Ma et al. / Displays 39 (2015) 68–79

defined arbitrary icons as icons that have no relation to their referent, such as the sign for a biohazard. Another proposition of icon taxonomy emerged in 1998 [18]. This proposition constructed a criterion system to classify icons by the means of the nature of sign, the arrangement of the signs and the modality. She figured out three kinds of icon with the name of concrete-icons, abstract-icons and symbolic icons. According to her theory, concrete-icons represent the objects as a photograph; while abstract-icons provide a perceptual relationship between object and concept. The first two categories were very much similar to Roger’s definition only using different terms of category name. However the implication of ‘‘symbolic icons” varied in two theories. Symbolic icons for Roger were those have abstract relationship between icon and representing object. To the contrary, Purchase categorized symbolic icon as the predefined convention that have no perceptional relationship between them. Lidwell’s study [19] on icon taxonomy took advantage of previous work but more precisely in categorization. They developed four types of icon based on the findings of Rogers. Icons named ‘‘similar, example, symbolic and arbitrary” were proposed. Similar icons visually represent an action, an object or a concept; while example icons exemplify or represent the common characters of a group of action, object or concept. Besides, symbolic icons were defined by the icons representing an action, object or concept ‘‘at a higher level of abstraction”. Finally arbitrary icons use the images with little or no relationship to the action, object or concept that are represented. Although researchers in above icon taxonomy studies announced different theory which they relied on, they were interested in the physical form of icon and how iconic representation expresses the real representing object [20,21]. Across various category titles applied in those classification methods, five possible relationships may occur between iconic representation and representing object, based on which some categories could be essentially considered into one [22]. Based on five relationships between icon and representing object, existing computer icons are able to be classified by physical appearance and applied according to real needs (see Fig. 1).

2.2. Icon taxonomy based on user’s perception Icon taxonomy based on physical appearance has been largely accepted and applied in the field of GUI. However, it would be found that the boundaries between five relationships in physical appearance were vague. Icons in one category may also fit the definition of another. For example, ‘‘representational icons” according to Lodding’s theory was divided into ‘‘resemblance icons” and ‘‘exemplar icons” under Rogers’ explanation. Thus icon taxonomy needs more dynamic classification criterion instead of rigid standard. Meanwhile, human cognitive behavior is more and more emphasized in interaction studies with the development of information technology and visualization science. Icon taxonomy based on user’s perception emerged and also play an important role in icon research. Studies of McDougall et al. [23,24] are representative in users’perception-based icon taxonomy. They proposed five criterions to classify computer icons: concreteness, semantic distance, familiarity, complexity and aesthetic appeal. According to the definition, concreteness implied the concrete degree of an icon representing the object in the real world; while semantic distance referred to the semantic difference between icon and its representing object. Concreteness and semantic distance are theoretically related to the criterion of icon taxonomy based on physical appearance, which revealed that the taxonomy investigated by McDougall was developed relied on previous studies instead of being completely independent. Apart from concreteness and semantic distance, other three criterions were new notions originating from users’ perception. Familiarity considers user’s cognitive understanding of icon and use frequency of iconic representation. The more frequently a user applies an icon or the representing object of the icon, the higher degree of familiarity this icon will have. This criterion revealed user’s empirical value produced to the physical appearance of an icon and it was interested as well in user experience. What’s more, familiarity influenced how user evaluates icon. For example, an experienced driver will quickly recognize that a road icon of exclamatory mark represents ‘‘aware of the danger”. When

Fig. 1. Icon taxonomy studies based on physical appearance [22].

X. Ma et al. / Displays 39 (2015) 68–79

familiarity is higher level, even an icon has no visually direct relation with the object user will get quick identification on iconic representation. It meant that physical appearance is no longer the only important criterion to classify icons compared with familiarity based on users’ perception. Complexity is a criterion with strong relation to users’ perception. The more complex an icon is, the more difficult it will be understood and reused by users in interface design. Research findings demonstrated that the effects of visual complexity appeared to be long lasting with differences in performance between simple and complex icons remaining even after considerable training [24]. Thus, user having different perception level will hold different points on the complexity of icon. It is partially associated with icon physical appearance but mainly depends on users’ own opinion. Aesthetic appeal was created from another cognitive series, however also related to other criterions and critical to evaluate an icon. A simple icon at a high level of concreteness and familiarity may easily arouse user’s aesthetic appeal. Among these criterions, aesthetic appeal is the most dependent with user’s personal perception. That is because even to the same icon, users will show different performance on aesthetic appeal and have different opinions to improve the design of icon. Thus achieving the perceptual consensus on aesthetic appeal of icons will more difficult than other criterions. The icon taxonomy method proposed by McDougall and her research team differed from that by physical appearance of icon. It focused more on the effect to icon categorization causing by users’ perception. However, some criterions mentioned in their studies still reflected the importance of the relationship between icon and its representing object, like concreteness and semantic distance. To the contrary, users’-perception-based icon taxonomy implied that physical appearance of icon was no longer the only standard to distinguish icons: users will provide different explanations onto the relation between an icon and its representing object; if user’s perception varies, various physical relations will be consequently produced.

2.3. Icon taxonomy based on representation strategies A deeper study on icon taxonomy has been recently carried out [25] and implied the diversity of classification criterion by representation strategies. It meant the taxonomy of computer icons cannot be accurately completed based on unique criterion but integrating from multiple views. Meanwhile one classification criterion may not be suitable to all kinds of icons. Each of these criterions has its own scope of application. This study took health care for an example. It was part of large research project in which they proposed to develop a computer application that automatically complemented patient instructions with icons. Researchers selected 846 icons from website and published journals, and tried to provide three criterions to group them at the same time. Icons were divided into lexical words (or content words) and function words (or grammatical words) according to lexical categorization of representing words, the first criterion. Lexical words included all content words like specific object, concept or action while function words referring to conjunction or proposition. Among 846 involved icons, only 14 of them represented function words. Besides, the findings showed that the majority of transitive verb and qualifier needed to be iconized by adding a noun instead of merely by itself. For example, thin is an adjective however cannot be represented directly by an icon but using ‘‘thin man” as a corresponding icon. The second criterion proposed in that study was designed for the icons of which the representing words were noun. This classification was carried out based on UMLS (Unified Medical Language


System). 562 noun icons were divided into entity (484 icons) and event (78 icons). The above two criterions were proposed by semantic analysis of representing words, where no direct relationship with icons was revealed. To the contrary, the third classification criterion implied a new notion: representation strategy, which reflected the path used to convert concepts into pictographs. Three sub-kinds of representation strategy were pointed out: visual similarity, arbitrary convention and semantic association. Particularly, visual similarity representation strategy was only employed to the icons of which representing objects were noun and entity. The first representation strategy was similar to ‘‘familiarity” mentioned by Mcdougall – icons were designed through the similarity between iconic symbol and represented object. This familiarity was able to be explained by either the familiarity onto icon itself or the relationship between icon and representing object. However this strategy worked merely when representing words were noun and when corresponding object had specific physical appearance. The second representation strategy was illustrated precisely by several conventions. Abstract convention was mainly geometric or verbal. For example arrows stood for ‘‘recycle” and the letter P for ‘‘parking”. Another convention was concrete convention which often had its original semantic reference. For example, a picture of skull represented ‘‘poison” and angel for ‘‘doctor”. When original semantic relationship faded out, this convention between icon and representing object remained. The last was transposed convention which no physical relationship existed between icon and represented object but referent. For example thumbs up represented ‘‘correct”. The third representation strategy differed from the first two. It linked semantically icon and representing object as a medium. Taking a clock and ‘‘time” as an example, there was no specific picture to correspond to ‘‘time”. However clock provided them with semantic bridge. Based on various modes of medium, researchers divided again the semantic association strategy into eight groups. The most typical groups were ‘‘exemplification” and ‘‘physical decomposition”. Exemplification icons were close to the example icons mentioned in physical-appearance-based classification. They took a representative symbol to reveal the represented object, such as Chinese cabbage for ‘‘food”. While physical decomposition icons were the images that implied a part of represented object, such as a microscope for ‘‘laboratory”. The common feature of two types of semantic association ‘‘exemplification” and ‘‘physical decomposition” was that the icon could be explained by the symbol in the picture or by the ensemble where symbol belonged to. For example, a picture of bicycle was able to represent ‘‘bicycle” as well as ‘‘transport”. Research findings showed that the three representation strategies integrated tightly, intercrossed and penetrated reciprocally. This study highlighted the influence of represented object when deciding which representation strategy would be applied. Also it did not always work to categorize an icon into a certain group. When representation strategy changed, the clusters within icons would as well vary, which did not depend on physical appearance of icon or users’ perception.

3. Proposed icon taxonomy based on application purpose in computer science The application of computer icons could trace back to visual design for computer interface. Taking use of these symbolic graphs users are able to easily master operations on the computer. Meanwhile computer interface turns to be more artistic and practical in both space and time compared with writing in text. However in


X. Ma et al. / Displays 39 (2015) 68–79

fact even in ancient Chinese characters and Mayan texts, icons were used to spread information and annotate knowledge [26,27]. These original symbols, not be called as icons, were oldest referent to represent information as a communication tool. In our daily life, road sign could be regarded as another special but useful kind of icon. They provide driving information and knowledge to drivers. Following the development of computer science, icons are also applied to map the knowledge in real world. The study of Lohse [28] showed that icon is one of visual knowledge representation. Different from other tools (graph, table, map, diagrams and network charts) icon illustrates a specific object with simple but clear image. Seen from these evidences, computers could not only work for operation guide but also knowledge transmission medium. Although existing icon taxonomy discussed above emphasizing the relationship between icon symbol and represented object, they did not systematically describe the difference on potential applied areas of icons. From physical-appearance-based icon taxonomy to representation strategy icon taxonomy, the boundary between icon categories blurs. How one icon is associated with one object may not be well explained by only one classification criterion. On the other hand, users’ perception starts to be more and more complex since more participants take place into icon design and icon cognition. Thus in this paper we propose a new way to classify computer icons with more focus on what this icon is designed for. Based on this criterion, two new terms are put forward to identify computer icons: action icon and knowledge icon. 3.1. Action icon We defined this type of computer icon as those guiding users to take specific operations on computer interface. Action icons are originally designed to provide visual instructions for users who are not familiar with the information technique they are using. They translate a long text description into several symbolic pictures. Users are supposed to click on an icon and complete certain task by computer. For example, we click capitalized b on the toolbar of Microsoft Word to make the word in bold. This action is explained by a ‘‘B” icon instead of writing ‘‘make the words in bold”. Action icons occupy a large part of computer interacting design because the initial objective for computer is to help humans do something faster and simpler. Users need action icons to learn how to employ the machines and complete the task step by step. Thus icons, no matter concrete or abstract, could suggest users ‘‘if you want to do this then click here” once the icon-operation relationship has been created. 3.2. Knowledge icon Differing from action icons, knowledge icons are those passing some information to users but not for action purpose. However this information does not aim to guide users do some operations step by step but to let users know something, especially complex or professional information, and finally form an information translating cycle ‘‘text-icon-text”. During or after this process, maybe users will do some actions, but not compulsory. Even producing actions, users may apply the knowledge for other purpose but not for operating on the computer. Knowledge icons are not rare in nowadays. For example, more and more medicine systems are applying icons to make convenient the searching task from millions of drugs descriptions. The use of knowledge icons reveals a high development of internet and network. We are not only interested in using computers to type a text by Microsoft Word but also in searching online useful information to improve the thesis writing. When information is limited online, spending a while to find a target and read it is an acceptable decision. To the contrary, in today’s

big data era, spending more time to find an item of information means missing a lot of other information. Thus knowledge icons could act as visual summary to illustrate the core points involved in these texts [29–31]. And then visual summary is able to be transferred in human’s brain as ‘‘what is telling by this text because it is explained by these icons”. This is what to be called by ‘‘text-icon-text” cycle. Knowledge icons could be regarded as medium between the information expressed online and the information got by users. Among these knowledge icons two kinds of them are frequently seen in computer interface applications: status icons and classification icons. Status icons are knowledge icons providing the current status information of given knowledge. For example an hourglass with different percentages of sand could represent the different processing stages of a project. While classification icons are those knowledge icons signifying the categorization information of knowledge items. Through common graphical features or related symbolic representation of icons, user is able to obtain the organization knowledge of represented objects. For example in the case of crisis management, classification icons may imply kinds of crisis and rescue methods that have already been structured. More details about how to apply these knowledge icons in computer interface will be precisely explained in the next section. Seen from these, the classification criterion of computer icons into action icons and knowledge icons is based on the objective to use the icons (what needing to gain from icons) and the possible way apply the icons (what users will do with the icons). In this classification, how iconic representation refers to the represented object is no longer considered. It is assumed to avoid the conflict from multiple criterions of icons classification based on physical appearance, user’s perception and representation strategy. In order to evaluate the notion of action icon and knowledge icon, a practical classification was carried out in computer icons application relying on this objective-oriented criterion.

4. Method and results To assess the icon taxonomy proposed in our study, we selected 620 icons from 140 icon-concerned applications presented recently (1994–2014) online from website, conference articles and journal articles. All these applications chose icons to improve theoretically or practically computer use in a specific background. The icons displayed in the following sections do not receive any change after being selected in our study. They keep their appearance in original applications. We worked out a two-steps method to classify these icons by our new criterion. At the first step 620 icons were divided into action icons and knowledge icons considering whether users would like to take use of the icon for further operation or for getting some knowledge. And then icons in each category will continue to be specified into more precise types according to the specific field they are applied. From selected 620 icons in total, 243 icons are action icons while 377 knowledge icons. The difference on the amount of two kinds of icons results mainly from the things they are representing. Most of the action icons try to visually illustrate general computer operations. Although these operations have unique features depending on application environment, they show a higher consistency with used design rules. For example, a cross often represents the operation ‘‘log out” or ‘‘close a webpage”. To the contrary, knowledge icons provide more possible iconic symbols due to various knowledge fields they may be involved. Thus compared to action icons, knowledge icons imply more creative designs. However, because applying icons for knowledge representation is in its infancy, the amount of knowledge icons is not significantly higher than that of action icons.

X. Ma et al. / Displays 39 (2015) 68–79

In the next two sections, we will explain the typical application fields where action icons and knowledge icons likely appear on the computer interface. From these evidences, the reason why to classify icons into action icons and knowledge icons will be clearer. 4.1. Action icon Introducing icons in user-interface design opened a new page in Human Machine Interface [32]. The features like being concise, intuitive, vivid allow icon acting as a communication tool and providing visual guide of operation. Icons explore a way to show the function of software or the dossier of desktop more effective. In spite of long textual descriptions, icons attract users’ attention as well as allow users to operate step by step. This function is meaningful to international working environment. Regardless which platform is used, icons provide a way to reach the object more convenient beyond language barriers [33]. Computer icons that are designed to graphically resemble their underlying functionality are better recognized by participants and may be also better recognized by users [34]. As a result, icon becomes general representation on the computer screen in several specific fields. Among 243 action icons found in the demonstrations the most largely occupied applications are software interface, web browser and mobile phone interface. The order of emerging action icons depends mainly on the development process of information technology and computer science. 4.1.1. Software interface Icon almost came into use synchronously when software was employed. They are displayed on the tool bar in the purpose to provide convenient path for the most frequently clicked operation. Instead of seeking from each drop-down menu, icons tell users what kind of function involved by this software through symbolic explanation. And it often helps especially when user is not familiar with the text on the menu. Text-free way allows icon making clearer the operations supplied by the software [35]. Text processing software is a typical icon-related application. General operations like switching words to bold and italics format, setting background color of text centring the selected text, all have its corresponding icon representation. User needs only click on them instead of seeking from textual menu. Words processing practice manifests that iconic access reduces significantly the time for completing text tasks [36]. Other types of particular-objective-based software are as well taking use of icon to visualize operation guide on the interface. Icon-based educational software design was an example [37]. The research explored how subjects in grade 1 (6–7 years old) and grade 3 (8–9 years old) identify auditory icons that are commonly introduced in educational software applications. The results indicated that the third-grade subjects were better equipped to identify auditory cues based upon two dimensions of interest than the first-grade subjects. Similar to picture book for children, icons have advantage to raise learners’ interest more than boring words. This is why more and more educational software applies icons on the interface. Through simple clicks, students could take opera-


tions and finally use the software to learn something, which is also mentioned in serious game interface studies [38]. Almost every kind of software is using icons as the component of interface nowadays. Although different types of software may have varied operations, icons make their interface more accessible especially for international user. Besides visual explanation of the functions involved in the software, icons are at the same time able to improve the aesthetical level of interface, which becomes a critical factor to evaluate. 4.1.2. Web browser Apart from software interface, web browser is another large icon-occurring application area [39]. Due to the development of Web and Network, web browser became a useful tool to get access of news on the Internet. More and more functions were created to improve user experience, such as saving favourite web content and navigating sites [40]. The use of icon on the web browser aims to provide visual bridge between symbolic buttons and possible operations of a web page. Even those who are not skilled in Internet surfing, like old people, may learn basic operations by iconic representation. Some studies investigated the appropriateness and effectiveness of icon design for Chinese web browser. The test website consisted of the design of a Chinese Operating System (COS) environment [41]. The COS was designed to be culturally rich, and have both visual and aural stimuli. They have pointed out that graphical user interfaces (GUIs), which include interactive images and animations, have opened a new dimension for visual languages and transformed symbolic systems into much more complex web applications. Besides this, IE7 takes as well attention in icons use. Some research has examined how icon design and location influence users when they surf the net with IE6 and IE7, and inspected how tabbed browsing would influence users when switching between sites [42] (see Fig. 2). Icons for web browser could be considered as a special case of icons for software. They are created to meet particular operation needs on the web where more fixed functions are defined. Unlike various software works for different purpose, there is more consistency across the various web browsers than across the various kinds of software, only differing in symbol choose, color and icon arrangement. 4.1.3. Mobile phone interface Applying icons on mobile phone and how icons affect mobile phone interface design is a popular topic studied in these years. The source of icon-based mobile phone menu may be traced back to the success of smart phone [43]. The current icon demonstrations on mobile phone interface could be divided into two aspects: menu design and application design. The menu design concerns how to create and display icons on the menu page in order to present on the whole the included functions of a mobile phone. For example, Fig. 3 shows one of the oldest icon-based mobile phone menu where simple and original symbols were chosen [44]. They were the initial icons representing limited phone operations like sending message, making a call or setting up

Fig. 2. Internet Explorer web browser interface for the Macintosh Version 5.2 [42].


X. Ma et al. / Displays 39 (2015) 68–79

Fig. 3. Icon-scenario based Animated Menu of mobile phone [44].

the time. Two main changes emerged into today’s phone menu: animated and personalized icon design. On one hand, varied dynamic displaying of phone icons implies current menu state and users are able to move and arrange them in the preferred order. On the other hand, difference between old and young [45], or different culture background is also discussed when icon menu is created. The symbols are becoming diversified for multi-culture cases and the color is brighter to get artistic visual experience for younger people [46]. Application design is another revolution for mobile phone interface [47]. Theoretically speaking, the use of icons for application design could be regarded as a similar attempt to software interface and web browser only changed from computer way into mobile way [48]. Since more and more users choose mobile phone to achieve their web needs, this trend leads to a higher attention on icon design for both software and web page on the phone [49,50]. Due to the limited size and resolution, icon of phone applications implies more advantage than text and requires more creative design. Through menu design and application design, icons for mobile phone interface act as a useful operation guide to ease daily phone use and improve the interaction between user, phone itself and the applications attached to it. 4.1.4. Brief summary Seen from the three most applied icon categories above, action icons are in fact the icons that were talked in general interface design studies. Since they occurred when GUI method was firstly pointed out, action icons have a long history and the majority of influential findings emerged in 20 centuries. No matter which specific goal they work for, software, web browser or mobile phone, action icons have the common objective to simplify the way of users’ operations on the machine. This is also a good illustration why they are called action icons and why it has a different applied objective from another kind of computer icon which will be explained in the next section. 4.2. Knowledge icon Lohse’s study [28] pointed out that unlike other types of visual knowledge representations (graphs, tables, maps, diagrams and network charts), icons have brief and clear imaged appearance to reach simple and unique representing objective. The protocol of knowledge icon began in early characters of Chinese and Maya [26,27]. They used iconic-like symbols to represent the things and communicate with each other. These symbolic characters could be considered as iconic annotation, by use of which people are able to interpret information to others with visual representa-

Fig. 4. Icon-based VCM system [55].

tions. From the icon resources involved in our studies, three main knowledge icon application fields were defined and illustrated as follows. 4.2.1. Healthcare Due to brief symbolic interpretation of represented object, iconic annotation has largely occurred in the domain of healthcare. People may be familiar with their body but may not be familiar with the name of each part of their body. Thus medical knowledge is often described in uncommon words and professional way, which is hard to be easily understood even for doctors. However, people are willing to learn some medical knowledge for keeping healthy and textual description is not accessible for their daily use. Resulting from the complexity of medical knowledge, iconic representation is a good way [51,52]. Although people may not heart the name of a part of the body before, they are familiar to the appearance and to the iconic representation of it. Thus explaining medical knowledge through an icon-mediated method is able to enhance collective participation in healthcare. A number of iconic applications for healthcare were carried out [53,54]. An icon system named VCM (visualization of medical knowledge) [55,56] was developed to simplify medicine searching from an online medical library. In the original application of VCM (focused on drug properties), the objective was not to find a document in an online library, but rather to find a paragraph in a (long) medical document (the drug’s Summary of Product Characteristics), each paragraph describing one drug property, such as a contraindication or an adverse effect. The interface is presented by combined icons, the graphical components representing respectively three knowledge categories: contraindication, drug interactions and drug adverse effects (see Fig. 4). Doctors will easily find target medicine by choosing the icons that have suitable components. The advantage of this icon-annotated medicine library is to save doctors’ time from searching with thousands of long textual descriptions. Iconic knowledge representation could be regarded as a bridge between medical text and patient (or doctor). They are able to not only better understand the knowledge, but also easily share it with others through a concrete translating language. 4.2.2. Crisis management Besides attempts in healthcare field, crisis management has also started to involve icons as annotation tool. The purpose of icon is to

X. Ma et al. / Displays 39 (2015) 68–79


Fig. 5. Iconic languages applied for semantic interoperability in emergency management [61].

represent visually kinds of crisis and tell people remember easily why this crisis happens and how to get rescue from it. Similar to medical cases, crisis management is close to our daily life however text-express way cannot attract people’s attention and well explain the implied knowledge. However iconic knowledge representation could reconstruct the scenes in daily life and point out accurately what is a possible crisis case. Collins’ fire safety symbols [57] were the earliest study on iconannotated crisis management. They have studied on twenty-five internationally proposed symbols for fire-safety alerting with other 91 U.S. participants. These symbols included three modes of symbol presentation (slides, placards, and booklets) and two modes of participant response (definition and multiple choice) as well as confidence ratings and production data (drawings). In this case, Researchers began to realize that icons had competence in guiding user to be informed directly with specific imaged representation. Although at the beginning the knowledge which icons referred to was displayed simply one after another in spite of being catalogued under a well-formed structure, people started to represent knowledge using image tools considering their uniform size and content. After Collins, several researchers tried to enrich the ‘‘icon warehouse” step by step. On one hand, they added more knowledge notions in crisis management and found corresponding icons to this new defined crisis cases or rescue methods. Like the study of NicIcon [58], which is to collect handwritten sketches containing iconic gestures and to access pen input recognition technologies for the domain of crisis management. Fitrianie and Rothkrantz [59] created as well a visual communication language for crisis management using icons. On the other hand, more possible icon formats were proposed with focus on icon design development [60,61] (see Fig. 5). Structured icon displaying way was more acceptable to visualize the relations between varied kinds of crisis cases. No matter which studied point was stated, all the icon-based crisis management applications emphasize the importance of map. It is due to the geographical feature of crisis and also partly a crucial factor of knowledge icons which will be discussed in the next section.

4.2.3. Online media management In order to enable the retrieval of visual knowledge, like image [62] or video, iconic annotations were also applied. Icons have common visual features with these special formats of knowledge

thus provide better applicability in these fields, which will be explained in details in Section 5. What’s more, these types of visual knowledge will be better annotated and shared by icons, since a long textual description may not briefly and precisely present their key content. Iconic knowledge representation is able to represent the whole file of knowledge or a part of character of the knowledge. For example, a photo of the view in a garden River could be annotated by an icon of symbolic design including flower, bridge and lawn. Meanwhile it could as well use a single flower icon to imply the objects in this photo. Thus the feature of knowledge icon for online image management mentioned here is related to the findings in several studies on visual retrieval of images [63]. Media Streams is one of the examples, which enables users to create multi-layered, iconic annotations of streams of video data [64]. Although some aspects of video can be automatically parsed, a sufficient representation requires that video be annotated. A Media Time Line enables users to visualize, browse, annotate, and retrieve video content. The challenges of creating a representation of human action in video are discussed in detail, with focus on the effect of the syntax of video sequences on the semantics of video shots. Iconic representation occurs also in music applications [65]. The icons are generated automatically, employing a neural net to determine the graphical parameters from acoustic features of the waveform stored in the audio files, which simplifies the music information retrieval [66].

4.2.4. Brief summary From the three icon domains mentioned above, knowledge icons have a particular objective to represent the information and knowledge. In other words, users are supposed to learn something from the visual illustration instead of taking actions on the icon. This is completely different from the objective of action icons. What has to be emphasized is that all these three domains reveal two kinds of knowledge icons pointed out in Section 3: status icons and classification icons. No matter in healthcare, crisis management or online media management, status icons and classification icons are possibly applied. For example, status icons could signify the stage of treatment or the status of crisis; while classification icons are able to clarify the style of music. The notion of ‘‘knowledge icon” could be regarded as a result of the development on knowledge engineering, an important branch in computer science. Although using icons to represent information is not a new term in graphical semiotic, applying them as


X. Ma et al. / Displays 39 (2015) 68–79

knowledge representation on the computer has not yet systematically studied. Unlike action icons, which have a long history and restrictive operation type, knowledge icon is young while having large applied area that knowledge in several fields is able to be iconized. Thus the three categories of knowledge icons listed before are only typical examples without covering all the cases. 5. Discussion Seen from the experimental results of icon taxonomy based on applying objective, action icon and knowledge icon are two meaningful terms to describe current icon applications in computer interface. Unlike previous three classification criterions, action icon and knowledge icon have a more certain boundary from different icon categories. No matter it is concrete icon or abstract icon, it will be called action icon if the applying objective is to let users do some operations on the computer. No matter it is exemplified icon or convention-based icon, it will be regarded as knowledge icon when applying objective is to help users understand and know something. Furthermore, though knowledge icons emerged later than action icons, they are having larger and larger applied stage in nowadays computer interfaces. As what has been mentioned before, the kind of computer operations is limited and most of them have traditional design template, which leads to the bottleneck of action icons. In contrast, knowledge icons are obtaining more focus as a result of centralized attention on online knowledge sharing. People in all kinds of applied areas have been thinking of better knowledge representation to make their own professional knowledge sharable to other specialists or normal users. Thus knowledge icon provides a visual way to explain and spread knowledge, at the same time knowledge turns to be simpler taking use of symbolic tools. In this case, we are not wondering ‘‘a little flower” is a concrete icon or an abstract icon, but thinking about what kind of knowledge could be represented by this icon. Seen from the reprehensive categories of action icon and knowledge icon, more details will provide on the details in the features of these two terms. Particularly, we will highlight the applied objective of icon and discuss the difference among existing classifying criterions, according to which when and how to use our proposed icon taxonomy will be well explained. 5.1. Difference from ‘‘verb–noun” criterion Classifying icons according to the language attribute of represented word was described in Nakamura’s study [25]. Before carrying out representation strategy classification, they stated an icon taxonomy method based on lexical words and function words. Particularly, among 832 lexical words 562 icons were noun and 157 were verb. The ‘‘verb–noun” criterion has some commons with action icon and knowledge icon. Both of these two criterions are interested in the difference between ‘‘dynamic” icons and ‘‘static” icons. ‘‘Dynamic” implies that the represented object is a process of action even the action is instantaneous like ‘‘beat” while ‘‘static” expresses the state or property of the represented object like ‘‘nature”. However, action icon and knowledge icon are not exactly what is explained by verb–noun criterion. First of all, the criterion purpose is varied. A verb icon or a noun icon is to distinguish the part of speech of corresponding represented word. To the contrary, an action icon or a knowledge icon is in order to tell what could be done with this icon later: to complete an action or to get some knowledge. Secondly, a verb icon is not always an action icon, and similarly a noun icon is not always a knowledge icon. For example, a pointing left arrow on the web browser is a noun icon because ‘‘arrow” is a noun, yet it is an action icon for the reason that users will click it to go back for former web page. Finally, an

icon is identified uniquely as a verb icon or noun icon, but it is whether an action icon or a knowledge icon needs to be discussed under additional condition. Verb–noun criterion decides an icon into verb or noun only according to the represented words. If it represents a noun it is a noun icon, otherwise it is verb icon. However, action icon and knowledge icon are two terms from objective-oriented classification. When objective changes, the icon category may also changes. This will be explained more precisely in the next section about applied environment of icons. 5.2. Relation with applied environment To identify whether an icon is an action icon or a knowledge icon has strong association with the environment where it is applied. Classifying icons into action icon and knowledge icon is an objective-oriented taxonomy method, which means the objective to apply this icon is one of crucial factors to determine the icon category it belongs to. Thus pointing out an action icon or a knowledge icon needs to consider why it is used in this place and then decide its attribute. In many cases, different applying purpose leads to different icon categories. For example, a paper of text is a regularly-seen icon in document editing software. This icon often represents the operation of writing or creating a new text. Thus it is an action icon in the objective to guide users ‘‘click here and start your writing task”. In contrast, a paper of text may also appear in knowledge organization system to define the attribute of an item of knowledge: it is an article rather than a photo or a piece of news. Users will get the information that this knowledge is illustrated by text, if they want to obtain image resources they need to keep on seeking. Under this condition, this icon turns to be a knowledge icon. However it may produce an argument about action icon. Sometimes we could also think that action icons as well provide essentially information to users, because it is the information that helps users to take the next operation. So are these action icons able to be regarded as knowledge icons? According to the definition of knowledge icons the answer is No. Actually, all the icons on computer interface try to provide information, the visual information. The key point to distinguish action icon and knowledge icon is what users want to do with this visual information. If they obtain the information in the purpose to know clicking on which icon the operation will be put forward, they are using action icon. If they need the information to know the details about an incomprehensible thing, knowledge icon is involved. Similarly, all the computer icons may lead to further actions. The only difference is the purpose of this action. For example, when a doctor clicks on the icons to search for an effective medicine in healthcare icon system, he is more interested in what will be found by clicking on this icon than in ‘‘this icon is clicked for searching task”. Thus they are using knowledge icon instead of action icon. This argument could be also revealed in file management icons. Using icon to manage files was one of the first successful applications for computer interface, which was earlier than knowledge icons. Based on the definition of action icons and knowledge icons, whether a computer icon is an action icon or a knowledge icon depends on what user will apply it to do. Thus in our opinion, it cannot simply saying file management icons are action icons or knowledge icons only when explicating the specific applied environment. For example, previous computer practitioners created various icons to separate different electronic files, like Microsoft word. The original purpose of these icons is help user to quickly find the files, read them or edit them. From this point of view, although these icons provide information on the type of files, they are developed for operation use. They are action icons. To the contrary, there has been as well increasing applications of file management icons in knowledge management. For example, depending on

X. Ma et al. / Displays 39 (2015) 68–79

kinds of knowledge carrier, such as report, news, article and photo, each item of knowledge needs also clarifying its file type. In this case, icons to represent each file type are knowledge icons because users are assumed to get some knowledge instead of doing further operations on them. Considered these arguments, file management icons could be reasonable as action icons or knowledge icons. When we explicitly refer to the icons for further reading and editing operations in the computer, they are action icons in line with the definition. Particularly, the fields where knowledge icons are preferred to be applied have some common features. On one hand, the knowledge involved in these areas is often professional and complex. Expressing by text may cause some understanding problem while icons could help to translate the knowledge in a visual way and in general cognition. Like knowledge on medicine and crisis management, iconic pictures will reveal long-text explanation through simple and concrete signification. Moreover, these application domains require often shorter response delay. For example, rescuers need to clarify as fast as possible what kind of disease happens and its severity in the case of crisis management. Knowledge icons could save their reading time on sent information making use of visual representation. This advantage compared to text on quicker knowledge seeking was manifested in previous studies of icons [67]. However it does not say that less professional knowledge cannot be represented by icons. To the contrary, knowledge more close to our daily life and needing no special comprehension, such as travelling or clothes, could be more interactive on the computer when represented by knowledge icons than by text. For example, an item of view spot followed by an icon of mountain and an icon of clock would tell this is a place for mountaineering and it has a long history. Touring enthusiasts who search information on the web will quickly get this knowledge instead of reading long description and become interested in this travelling system. On the other hand, most of the fields applying knowledge icons may have a special focus on location information. For example, healthcare is related to which part of human body needing treated while crisis management focuses as well on the patients’ and rescuers’ real-time location. Knowledge icons could help in visually representing both ‘‘what” and ‘‘where” information through their symbols and the place at which they are put. Since icon is easier to be marked on the geographical map compared to text, it is assumed that knowledge icons are better to represent knowledge relationship when regarding a corpus of knowledge as a concept map. This is also why icons are applied by online map to tag found information [68]. However it is not to say knowledge icons cannot be applied in other domains, but when we emphasize location information, knowledge icons are better choices than text. From these evidences, knowledge icon serving for map-involved knowledge management is a choice decided by its own characteristics and the interactive needs of given knowledge. Consequently, unlike classifying icons based on physical appearance may cause diversity due to blur boundary between concrete and abstract, action icons and knowledge icons rely on the applying environment. This is one of the reasons that calling this method as objective-oriented taxonomy. The objective to employ an icon in certain application represents the role it will play though they always designed for interface improving. Especially in some cases, such as knowledge organization system, action icons and knowledge icons are able to co-exist. Knowledge icons represent what kind of knowledge could be found and action icons tell how to make this knowledge be found. 5.3. Difference and connection with former icon taxonomy Three studied icon taxonomy criterions have been reviewed in the second section: physical-appearance-based, user’s-


perception-based and representation-strategy-based. The objective-oriented classifying criterion is not independent to them but have some connection. First of all, objective-based icon taxonomy is exploration of former ones. Action icon and knowledge icon do not have either conflict with previous icon types or adding more precise categories based on those classification theories. However our proposed icon taxonomy studied computer icons from a different point of view. The three existing criterion focused more on characters of icons themselves. For example, physical appearance concerns how graphical or symbolic characters express the semantics of represented object; while user’s perception implies human’s evaluation on these characters. Representation strategy could be regarded as an integrated version of various physical-appearance-based criterions. Particularly, objective-oriented criterion pays emphasis on the applied purpose of icons. We are only interested in what kind of interface this icon is able to be used for, no matter it is a concrete icon or an abstract icon. Thus the focus point is no longer a group of isolated, static icons but a system including computer icons and all the applications with visual interaction needs. How to classify these icons requires as well considering the applied environment apart from their own represented words. Secondly, former icon taxonomy is also acceptable to action icon and knowledge icon. Whether it is action icon or knowledge icon, designers need thinking about what kind of representation will better represent the object. This creation rule reveals the principle of physical appearance and user’s perception criterion. How to translate the relation between icon and represented object, and how to seize user’s cognitive regularity is similarly the design problem for action icon and knowledge icon. However, generally abstract icons will appear more frequently in knowledge icons due to the features of represented words. Action icons express the operations that are usually a series of specific actions on the machine. After practicing for some time users could be familiar to these icons and to what is represented by them. To the contrary, a large part of knowledge is abstract concept and it is hard to find an accurate corresponding symbol to represent it. Thus some knowledge icons use abstract icon to make convention between icon and represented object. That is also one of the reasons why action icon is developed earlier and faster than knowledge icon. Thirdly, these kinds of icon taxonomy are complement to each other. Physical appearance criterion is the most direct and comprehensible way but may cause blur boundary between different icon categories and confusion between icon taxonomy based on different terminologies they used. User’s perception criterion enhances user’s participation in interface design however too subjective. Representation strategy explains completely the relation between icon and represented object yet it is complex to form creation rule. Meanwhile, objective-oriented criterion considers dynamically the applied environment and highlights the importance of iconic knowledge representation while at the same time it replied on the findings in other icon taxonomy. In a result, objectiveoriented icon taxonomy improves multi-theory problem in previous icon taxonomy and provides a larger stage for icon applications since the development of icon is attached to that of interaction design. Designers are required to create icons specifically for given interface instead of reusing icon resources.

6. Conclusions This paper presents an objective-oriented icon taxonomy method and defines two new terms of computer icon as action icon and knowledge icon. The practice of icon classification manifested that this kind of icon taxonomy focuses less on the relation between icon and its representing object, which simplifies the


X. Ma et al. / Displays 39 (2015) 68–79

problems on blur boundaries of physical-appearance-based method. The sub-categories of this icon taxonomy suggest the applied preference of each potential field and make clearer the development path from action icon to knowledge icon. Meanwhile, the notion of knowledge icon is meaningful to today’s interface design and in line with attention on online knowledge sharing. Along with other proposed icon taxonomies, our result is expected to contribute to icon system construction not only for operation visualizing but also for knowledge-centralized interaction. Acknowledgements This research is sponsored by the National Natural Science Foundation of China (71403201), Social Sciences Foundation of Shaanxi Province of China (2014L03), the MOE Project of Scientific Research Foundation for the Returned Overseas Chinese Scholars ([2014]1685) and the Fundamental Research Funds for the Central Universities (JB140611). References [1] T. Short, Peirce’s Theory of Signs, Cambridge University Press, 2007. [2] Apple Computer, Inc., Macintosh Human Interface Guidelines, Addison Wesley, Publishing Company, 1992. [3] A. Paivio, Mental Representations: A Dual Coding Approach, Oxford, Oxford University Press, England, 1986. [4] C. Peirce, in: C. Hartshorne, P. Weiss, A.W. Burks (Eds.), Collected Writings, vol. 8, Harvard University Press, 1931-58. [5] P. Barr, J. Noble, R. Biddle, Icons R. Icons, in: The Fourth Australasian User Interface Conference, Adelaide, Australia, Conf. Res. Pract. Inf. Technol., vol. 18, 2006. [6] W.K. Horton, The Icon Book: Visual Symbols for Computer Systems and Documentation, John Wiley & Sons, New York, 1994. [7] S. McDougall, S. Isherwood, What’s in a name? The role of graphics, functions, and their interrelationships in icon identification, Behavior Research Methods 41 (2) (2009) 325–336. [8] A. Marcus, Icon and symbol design issues for graphical user interfaces, in: E.M. del Galdo, J. Nielsen (Eds.), International User Interfaces, John Wiley & Sons, New York, 1996, pp. 257–270. [9] W.L. Martinez, Graphical user interfaces, WIREs Comp Stat 3 (2011) 119–133, [10] J. Rosenblatt, Former Apple Designer Kare Testifies at Samsung Patent Trial, Business week, 2012. [11] K.N. Lodding, Iconic interfacing, IEEE Comput. Graphics Appl. 3 (2) (1983) 11–20. [12] D. Gittins, Icon-based human–computer interaction, Int. J. Man Mach. Stud. 24 (6) (1986) 519–543. [13] W. Gaver, Auditory icons: using sound in computer interface, Human– Computer Interact. 2 (2) (1986) 167–177. [14] G. Lindgaard, J. Chessari, E. Ihsen, Icons in telecommunication: what makes pictorial information comprehensible to the users?, Aust Telecommun. Res. 21 (2) (1987) 17–29. [15] Y. Rogers, Icons at interface: their usefulness, Interact. Comput. 1 (1) (1989) 105–117. [16] M.A. Blattner, D.A. Sumikawa, R.A. Greenberg, Earcons and icons: their structure and common design principles, Human–Computer Interact. 4 (1) (1989) 11–40. [17] J.M. Webb, P.E. Sorenson, N.P. Lyons, An empirical approach to the evaluation of icons, SIGCHI Bull. 21 (1) (1989) 87–90. [18] H. Purchase, Defining multimedia, IEEE Multimedia 5 (1) (1998) 8–15. [19] W. Lidwell, K. Holden, J. Butler, Universal Principles of Design, Rockport Publishers, Massachusetts, 2003. [20] M.E. Familant, D.C. Detweiler, Iconic reference: evolving perspectives and an organizing framework, Int. J. Man Mach. Stud. 39 (1993) 705–728. [21] Y. Yeh, D. Lee, Y. Ko, Color combination and exposure time on legibility and EEG response of icon presented on visual display terminal, Displays 34 (1) (2013) 33–38. [22] H. Wang, S. Hung, C. Liao, A survey of icon taxonomy used in the interface design, in: Proceedings of the ECCE 2007 Conference, August 28–31, 2007, pp. 203–206. [23] S.J.P. McDougall, O. de Bruijn, M.B. Curry, Exploring the effects of icon characteristics on user performance: The role of icon concreteness, complexity, and distinctiveness, J. Exp. Psychol. Appl. 6 (2000) 291–306. [24] S. Isherwood, Graphics and semantics: the relationship between what is seen and what is meant in icon design, in: Proceedings of HCII 2009, LNAI, vol. 5639, pp. 197–205. [25] C. Nakamura, Q. Zeng-Treitler, A taxonomy of representation strategies in iconic communication, Int. J. Hum Comput Stud. 70 (2012) 535–551. [26] J.F. Sowa, Signs, processes and language games: foundations for ontology, 2006. .

[27] P. Honeywill, A comparison between maya hieroglyphs and computer icons, AI & Soc 14 (1999) 395–410. [28] J. Lohse, H. Rueter, K. Biolsi, N. Walker, Classifying visual knowledge representations: a foundation for visualization research, in: IEEE Computer Society Technical Committee on Computer Graphics, Siggraph: ACM Special Interest Group on Computer Graphics and Interactive Techniques, ISBN: 0-8186-2083- 8, 1990, pp. 131–138. [29] X. Ma, J.-P. Cahier, Semantically structured VDL-based iconic tags system, in: Proceedings of 15th International Conference on Human–Computer Interaction, July 21–26 2013, Las Vegas, USA, Lecture Notes in Computer Science Series, vol. 8016, pp. 465–474. [30] X. Ma, J.-P. Cahier, An exploratory study on semantic arrangement of VDLbased iconic knowledge tags, Knowledge Org. 41 (1) (2014) 14–29. [31] X. Ma, J.-P. Cahier, Graphically structured icons for knowledge tagging, J. Inf. Sci. 40 (6) (2014) 779–795. [32] R. Nasanen, H. Ojanpaa, Effect of image contrast and sharpness on visual search for computer icons, Displays 24 (3) (2003) 137–144. [33] S. Passini, F. Strazzari, A. Borghi, Icon-function relationship in toolbar icons, Displays 29 (5) (2008) 521–525. [34] J. Ferreira, J. Noble, R. Biddle, A case for iconic icons, in: The Seventh Australasian User Interface Conference, Hobart, Australia, Conf. Res. Pract. Inf. Technol., vol. 50, 2006. [35] F. Shipman, R. Airhart, H. Hsieh, Visual and spatial communication and task organization using the visual knowledge builder, in: Proceedings of the 2001 International ACM SIGGROUP Conference on Supporting Group Work, 2001, pp. 260–269 (ISBN:1-58113-294-8). [36] R. Tanya, P. Beelders, J. Blignaut, T. McDonald, E. Dednam, The impact of different icon sets on the usability of a word processor, Lect. Notes Comput. Sci. 4559 (2007) 250–257. ISSN 0302-9743. [37] J.A. Jacko, The identifiability of auditory icons for use in educational software for children, Interact. Comput. 8 (2) (1996) 121–133. [38] G. Vardaxoglou, E. Baralou, Developing a platform for serious gaming: open innovation through closed innovation, in: 4th International Conference on Games and Virtual Worlds for Serious Applications, Proc. Comput. Sci., vol. 15, 2012, pp. 111–121. [39] S. Maiti, D. Samanta, S. Ranjan Das, M. Sarma, Language independent iconbased interface for accessing internet, communications in computer and information, Science 191 (2011) 172–182. [40] T. Lindberg, R. Näsänen, The effect of icon spacing and size on the speed of icon processing in the human visual system, Displays 24 (2003) 111–120. [41] S. Shen, S.D. Prior, K. Chen, M. You, Chinese web browser design utilizing cultural icons, Lect. Notes Comput. Sci. 4560 (2007) 249–258. [42] J. Lin, M. Hsieh, H. Yu, P. Tsai, W. Shiang, Comparing the usability of the icons and functions between IE6.0 and IE7.0, Lect. Notes Comput. Sci. 5610 (2009) 465–473. [43] C. Gatsou, A. Politis, D. Zevgolis, From icons perception to mobile interaction, in: 2011 Federated Conference on Computer Science and Information Systems (FedCSIS), 2011, pp. 705–710. [44] C. Lim, A case study of icon-scenario based animated menu’s concept development, in: Proceedings of the 8th Conference on Human–Computer Interaction with Mobile Devices and Services, Helsinki, Finland, 2006, vol. 159, pp. 177–180 (ISBN: 1-59593-390-5). [45] T. Lindberg, R. Nasanen, K. Muller, How age affects the speed of perception of computer icons, Displays 27 (4–5) (2006) 170–177. [46] P. Galdon, R. Madrid, E.J. de la Rubia-Cuestas, A. Diaz-Estrella, L. Gonzalez, Enhancing mobile phones for people with visual impairments through haptic icons: the effect of learning processes, Assist. Technol. 25 (2) (2013) 80–87. [47] J. Watanabe, I. Nakajima, Moving-icon-based GUI for accessing contents at ease on mobile phones, Int. Conf. Consumer Electron. (2005) 431–432. [48] W. Park, S. Han, Intuitive multi-touch gestures for mobile web browsers, Interact. Comput. 25 (5) (2013) 335–350. [49] Y. Salman, H. Cheng, P.E. Patterson, Icon and user interface design for emergency medical information systems: a case study, Int. J. Med. Informatics 81 (1) (2012) 29–35. [50] S. Ghayas, S. Sulaiman, M. Khan, J. Jaafar, Qualitative study to identify icons characteristics on mobile phones applications interfaces, IEEE Sympos. Wireless Technol. Appl. 2013 (2013) 310–315. [51] D. Gotz, J. Sun, N. Cao, Multifaceted visual analytics for healthcare applications, IBM J. Res. Dev. 56 (5) (2012), JRD.2012.2199170. [52] J. Cowgill, J. Bolek, Symbol Usage in Health Care Settings for People with Limited English Proficiency: Part One: Evaluation of Use of Symbol Graphics in Medical Settings, Robert Wood Johnson Foundation, Scottsdale, AZ, 2003. [53] P.S. Houts, R. Bachrach, J.T. Witmer, C.A. Tringali, J.A. Bucher, R.A. Localio, Using pictographs to enhance recall of spoken medical instructions, Patient Educ. Couns. 35 (2) (1998) 83–88. [54] L.N. Ngoh, M.D. Shepherd, Design, development, and evaluation of visual aids for communicating prescription drug instructions to non-literate patients in rural Cameroon, Patient Educ. Couns. 30 (3) (1997) 257–270. [55] J.B. Lamy, C. Duclos, A. Bar-Hen, P. Ouvrard, A. Venot, An iconic language for the graphical representation of medical concepts, BMC Med. Inform. Decis. Mak. (2008). [56] J.H. Vincent, A.E. Bandrowski, A.-S. Pepin, B.J. Gonzalez, A. Desfeux, OMIC tools: an informative directory for multi-omic data analysis, Database (2014), http:// tent/2014/bau069.long.

X. Ma et al. / Displays 39 (2015) 68–79 [57] B.L. Collins, N. Lerner, Assessment of fire safety symbols, Hum. Factors 24 (1) (1982) 75–84. [58] R. Niels, D.J.M. Willems, L.G. Vuurpijl, The NicIcon database of handwritten icons, in: Proceedings of ICFHR 2008, Montreal, Canada, August 19–21, 2008. [59] S. Fitrianie, L. Rothkrantz, A visual communication language for crisis management, Int. J. Intell. Cont. Syst. (Spec. Issue Distrib. Intell. Syst.) 12 (2) (2007) 208–216. [60] X. Ma, N. Matta, M. Sediri, J.-P. Cahier, VDL-based Iconic Co-annotation in Crisis Management, CTS 2014, Minneapolis, Minnesota, USA, May 19–23, 2014. [61] R. Peter, J.W. Van Aalst, F. Wilson, T. Hofmann, Using icons as a means for semantic interoperability in emergency management, in: 8th International Conference on Geographic Information for Disaster Management, Enschede, NI, December 13–14, 2012. [62] R. Lam, K. Cheung, H. Ip, L. Tang, R. Hanka, An iconic and semantic content based retrieval system for histological images, in: 4th International Conference on Advances in Visual Information Systems, Lyon, France, November 2–4, 2000, Lect. Notes Comput. Sci., vol. 1929, pp. 384–395.


[63] J. Lim, J. Jin, Image retrieval using spatial icons, in: IEEE International Conference on Multimedia and Exp, vol. 1–3, Taipei, TAIWAN, June 27–30, 2004, pp. 1615–1618. [64] M. Davis, Media Streams: an iconic visual language for video annotation, Telektronikk 93 (4) (1993) 59–71. [65] P. Kolhoff, J. Preuss, J. Loviscach, Content-based icons for music files, Comput. Graph. 32 (5) (2008) 550–560. [66] W. Machida, T. Itoh, Lyricon: a visual music selection interface featuring multiple icons, in: 15th International Conference on Information Visualisation, 2011, pp. 145–150. [67] X. Ma, N. Matta, J.-P. Cahier, Y. Cheng, How to visually access knowledge using VDL-based iconic tags, in: ACM Conference on Human Factors in Computing Systems (CHI2015). [68] X. Zhang, C. Liao, Y. Liu, J. Li, An icon map-based exploratory analytical approach for multivariate geospatial data, Sci. China – Inf. Sci. 56 (1) (2013).