CONSCIOUSNESS AND COGNITION ARTICLE NO.
6, 87–107 (1997)
Evaluative Learning with ‘‘Subliminally’’ Presented Stimuli Jan de Houwer, 1 Hilde Hendrickx, and Frank Baeyens University of Leuven, Leuven, Belgium Evaluative learning refers to the change in the affective evaluation of a previously neutral stimulus (NS) that occurs after the stimulus has been associated with a second, positive or negative, affective stimulus (AS). Four experiments are reported in which the AS was presented very briefly. Significant evaluative learning was observed in participants who did not notice the presentation of the affective stimuli (ASi) (Experiment 2) or could not discriminate between the briefly presented positive and negative ASi when asked to do so (Experiment 3). In two other experiments (Experiments 1 and 4), no significant learning effect was obtained. A meta-analysis performed on the present and previously reported results (De Houwer, Baeyens, & Eelen, 1994) gave evidence for a small, though statistically reliable evaluative learning effect when ASi are presented ‘‘subliminally.’’ This finding supports the hypothesis that evaluative associations can be learned implicitly. 1997 Academic Press
Research on evaluative learning has shown that under certain conditions, the affective connotation of a neutral stimulus (NS) will change if it is repeatedly presented together with a second, affectively relevant (positive or negative), stimulus (affective stimulus: AS). Associative evaluative learning is a robust phenomenon that can be demonstrated with a large variety of procedures and stimuli. For instance, it has been found with pictures of human faces (e.g., Baeyens, Eelen, Crombez, & Van den Bergh, 1992), paintings (e.g., Levey & Martin, 1975), and statues (Hammerl & Grabitz, 1993), with words (Jaanus, Defares, & Zwaan, 1990; Staats & Staats, 1957), gustatory stimuli (Baeyens, Eelen, Van den Bergh, & Crombez, 1990a; Zelner, Rozin, Aron, & Kulish, 1983), and even brand names (Stuart, Shimp, & Engle, 1984). The following procedure is commonly used (e.g., Baeyens et al., 1992; Hammerl & Grabitz, 1993; Levey & Martin, 1975). At the beginning of the experiment, participants are asked to affectively evaluate a set of pictures. Afterward, pictures which were given a neutral rating (NSi) are assigned to either a liked or disliked picture (ASi). During a learning phase, each NS–AS pair is presented for a certain number of times. As a part of a larger cover story, participants are asked to look attentively at the pictures. The experimenter does not mention that each NS is only paired with one AS. After the learning phase, participants are asked to evaluate the stimuli according to how they feel about them at that moment in time. It is typically found that the liking of NSi paired with a positive AS increases, whereas the liking of NSi paired with a negative AS decreases. A remarkable feature of evaluative learning is that the evaluative changes seem to occur independently of what participants consciously know about the crucial NS– 1 To whom reprint requests should be addressed at Department of Psychology, University of Leuven, Tiensestraat 102, B-3000 Leuven, Belgium. E-mail: [email protected]
87 1053-8100/97 $25.00
Copyright 1997 by Academic Press All rights of reproduction in any form reserved.
DE HOUWER, HENDRICKX, AND BAEYENS
AS associations. In other words, evaluative associations can be learned implicitly (Baeyens & De Houwer, 1995; Baeyens, De Houwer, & Eelen, 1994). This conclusion is supported by two groups of studies. First, several dissociations have been found between the number of NS–AS associations participants are aware of and the degree of evaluative learning that is observed2. In a number of studies it was found that the size of individual evaluative learning effects is not correlated with the number of NS–AS associations a participant is aware of (Baeyens, Crombez, Van den Bergh, & Eelen, 1988; Baeyens, Eelen, & Van den Bergh, 1990b; Baeyens et al., 1992; Baeyens, Hermans, & Eelen, 1993). Related to this it was found that participants who did not know any of the relevant associations showed the same degree of evaluative learning as those who did know some of the associations (Baeyens et al., 1990b). Baeyens et al. (1990b) also observed that instructing participants to search for the NS–AS associations during the learning phase dramatically increased the number of associations participants are aware of, but had no influence on evaluative learning (Baeyens et al., 1990b). Another study (Baeyens, Eelen, Van den Bergh, & Crombez, 1989) showed that perceptual similarity between the NS and AS of a pair significantly enhanced awareness of the NS–AS associations but did not affect evaluative learning. Finally, Baeyens et al. (1992) showed that increasing the number of NS–AS pairings from 2 to 5 and from 5 to 10 increased both evaluative learning and awareness of the associations. However, when the number of pairings was further increased from 10 to 20, evaluative learning decreased, whereas awareness kept increasing (Baeyens et al., 1992). The hypothesis that evaluative associations can be learned implicitly is also supported by a second group of studies in which the ASi were presented ‘‘subliminally’’3. During a learning phase, Niedenthal (1990) presented slides showing a novel cartoon character (NS). For some participants, these slides were preceded by a very brief presentation of a face (AS) expressing joy, for other participants the face expressed disgust, and for a third group the face had a neutral expression. When cartoon slides were preceded by an undetected face expressing disgust, participants afterward ascribed more negative traits to the cartoon character than participants in the joy condition. Krosnick, Betz, Jussim, and Lynn (1992) presented slides of a person going about normal daily activities (NS). As in Niedenthal’s (1990) study, each slide was preceded by a briefly presented affect-arousing picture (AS). However, instead of pictures of faces, Krosnick et al. (1992) used pictures of more complex scenes (e.g., a pair of kittens or a bucket of snakes). For half of the participants, the briefly presented affect-arousing slides showed a positive event, whereas for the other participants the slides showed a negative event. During the testing phase, the first group 2 In these studies, conscious knowledge of NS–AS associations was assessed by asking participants to indicate for each NS the AS with which it was paired or to guess whether the NS was paired with a positive, negative, or neutral AS. A participant was regarded as aware of a particular NS–AS association if (s)he (a) indicated the AS that had been paired with the NS, (b) indicated an AS that did not actually follow the NS, but which had the same valence as the crucial AS, or (c) when the participant correctly verbalized the valence of the AS. 3 The word ‘‘subliminally’’ is put between quotation marks because of its controversial and ill-defined meaning. Throughout the paper, we use the phrase ‘‘subliminally presented’’ to refer to brief presentation durations which are too short to allow for a subjective experience of the presented stimuli.
IMPLICIT EVALUATIVE LEARNING
of participants expressed more positive attitudes toward the person shown in the learning phase than the second group, despite the fact that none of the participants had detected the affect-arousing pictures during the learning phase. In a study conducted at our laboratory, we (De Houwer, Baeyens, & Eelen, 1994) used existing words as NSi and ASi instead of pictures. For each participant, 12 neutral words (NSi) were paired twice with a briefly presented and heavily masked positive word and 12 other words were paired with a briefly presented negative word. Afterward participants rated originally neutral words as more positive when they were paired with positive words than with negative words, despite the fact that they had not noticed the presentation of the affective words. Experiments with subliminally presented ASi probably provide the most compelling evidence for implicit learning of evaluative associations. By presenting the ASi subliminally, it becomes impossible for participants to consciously detect NS–AS associations because this requires that both elements of the association are consciously identified. So instead of directly assessing whether the associations themselves have been consciously detected, it is assessed whether a precondition for consciously detecting associations is met, namely, the conscious identification of both elements of the association. Note that this is a very conservative approach, because conscious identification of the elements of an association is, although necessary, by no means a sufficient condition to consciously identify the association itself. This approach to the study of implicit learning of evaluative associations requires a criterion for determining whether or not the elements of the association have been consciously identified. However, because ‘‘consciousness’’ is an ill-defined concept, opinions differ as to what constitutes the correct criterion for determining conscious perception. Some authors (e.g., Merikle & Cheesman, 1986; Baeyens et al., 1994) argue that consciousness is ultimately linked to subjective experience and that participants can be considered as being conscious of stimuli or information if they report to have experienced the presence of the stimuli or information. Other researchers (e.g., Holender, 1986) defend the position that more objective, performance measures have to be used if one wants to conclude that the presentation of a stimulus was ‘‘truly’’ subliminal. We believe that this difficult issue is not problematic when the aim is to study whether (evaluative) associations can be learned implicitly. For this purpose, it is sufficient to assess whether or not participants have subjectively experienced the presentation of the stimuli: If participants have not subjectively experienced the ASi that were presented during the learning phase, how could they then have consciously identified NS–AS associations? In our primary study on evaluative learning with subliminally presented stimuli (De Houwer et al., 1994), such a subjective criterion of conscious perception was used. At the beginning of the experiment, participants were told that words would be presented and that each word would be immediately preceded and followed by a row of X’s. Participants were not informed about the presence the ASi which were briefly presented after each clearly visible neutral word and before the second row of X’s. In order to check whether participants consciously noticed the presentation of the briefly presented ASi, they were asked two questions at the end of the experiment. First, it was asked whether they had noticed anything strange during the experiment, something the experimenter had not mentioned. Next they were asked whether
DE HOUWER, HENDRICKX, AND BAEYENS
they had seen that immediately after each clearly visible neutral word and before the second row of X’s, another word was briefly presented. Only participants who gave a negative answer to both questions (or who only mentioned some irrelevant aspect of the procedure) were regarded as unaware. When participants were told that neutral words were immediately followed by briefly presented positive or negative words, they typically were very surprised. In our opinion, it is more than reasonable to assume that these participants could not have consciously identified NS–AS associations. This conclusion holds regardless of whether one would regard the ASi presentations as ‘‘truly’’ subliminal or not. Although we believe that an adequate assessment of what participants have subjectively experienced is sufficient in this context, other, more objective, measurements can be and have been used. For instance, Krosnick et al. (1992) conducted a pilot study in which stimuli were presented in the same way as in their actual experiment. In the pilot study, however, participants were informed that ASi would be presented, and were asked on each trial to guess whether the briefly presented AS had a positive or a negative valence. Krosnick et al. observed that participants’ discrimination performance did not exceed chance levels. The fact that participants could not discriminate above chance when asked to do so strongly suggests that the ASi were presented too briefly to allow for conscious identification, which in turn suggests that the evaluative learning effects observed in the actual study were due to implicit learning. Therefore, instead of directly asking participants whether or not they have noticed the presentation of ASi, one can also indirectly infer that the ASi could not have been noticed during the crucial learning phase because (other) participants could not even determine the valence of the ASi when asked to do so. Valence-discrimination performance thus provides an alternative way to assess whether or not learning was implicit. Additionally, by assessing performance on a valence-discrimination task, one can also answer a second, partially independent question: Can evaluative associations be learned even if the affective connotation of the ASi can not be consciously detected? Both issues are independent because knowing that evaluative associations can be learned implicitly does not imply that they can be learned when the affective connotation of the ASi can not be detected. The main aim of the present research was to further examine whether evaluative associations can be learned implicitly. As was mentioned before, studies with subliminally presented stimuli constitute a very conservative test of this hypothesis and could thus provide the most direct and compelling evidence for implicit evaluative learning. However, the evidence obtained until now is scarce and the evaluative learning effects with subliminally presented ASi that have been reported are typically small and often just barely significant. Moreover, the studies of Niedenthal (1990) and Krosnick et al. (1992) used a between-participants design which did not control for possible nonassociative mood effects induced by the subliminally presented stimuli, although Krosnick et al. did control for mood effects in an indirect manner. In order to test the reliability of evaluative learning with subliminally presented ASi, four experiments were conducted, all using a procedure similar to the one used in our primary experiment (De Houwer et al., 1994). In all experiments, a subjective criterion for conscious perception was used. In two experiments, a valence-discrimination measure was used in addition to the subjective measure. Whereas Krosnick
IMPLICIT EVALUATIVE LEARNING
et al. (1992) assessed valence-discrimination performance in an independent group of participants, in the present experiments, a valence-discrimination task was given to those participants who participated in the actual learning experiment. An obvious advantage of this approach is that it provides the opportunity of determining for each participant in the actual learning experiment whether or not discrimination performance was different from chance. There are two important reasons why both measures were used. First, it allowed us to test the hypothesis of implicit evaluative learning in an even more conservative and stringent way: All participants for whom there is even the slightest direct (subjective experience) or indirect (above chance valence-discrimination) hint that they may have detected ASi (and who thus might have consciously detected NS–AS associations) will be excluded from the analyses. Second, by using a valence-discrimination measure it can be examined whether participants can (implicitly) learn evaluative associations even if they are not able to discriminate between positive and negative ASi. EXPERIMENT 1
Experiment 1 was an exact replication of our primary experiment (De Houwer et al., 1994), except that in addition to a subjective awareness measure, a valencediscrimination measure was used. Method Research participants. Fifty-seven first-year psychology students participated in order to partially fulfil course requirements. As in all following experiments, participants had normal or corrected to normal vision and were native Dutch speakers. Materials. The same stimuli were used as in our primary study (De Houwer et al., 1994). In that study, 24 affectively neutral, 12 negative, and 12 positive existing Dutch words were selected based on a pilot study. On a scale from 210 (very negative) to 110 (very positive), the mean rating of the selected neutral, negative, and positive words was 20.04, 24.37, and 15.18 respectively. t Tests showed that all means differed significantly (p , .001 for all comparisons). The 24 neutral words were then divided into two lists which were again identical to the lists used in our primary study. Lists were constructed in such a way that they did not differ in the mean evaluation of the words [F(1, 23) , 1]. The NSi and ASi that were used can be found in Appendix 1. Stimuli were presented on an 70-Hz SVGA color monitor connected to an IBM-compatible 486 computer. Words were presented in white uppercase letters on a black background. The presentations were controlled by a Turbo Pascal 5.0 program which operated in text mode. The program that controlled the learning and testing phase was the same as the one used by De Houwer et al. (1994). During the valence-discrimination task, a second program was used. Both programs contained an adapted CRT unit which allowed to operate with several visual pages that could be switched immediately. The experiment was run in a dimly lit room. Lighting conditions and intensity of the presented stimuli remained stable throughout the entire experiment. Procedure. Participants were told that the experiment concerned the semantic processing of words. It was said that they would see two blocks of 24 words and that
DE HOUWER, HENDRICKX, AND BAEYENS
each word would be preceded and followed by a row of X’s. They were instructed to look attentively at the screen from the moment the first row of X’s appeared until the moment that the second row of X’s disappeared. This was said to be important because all words would then be clearly perceived for a period that was exactly the same for all words. No mention was made about the subliminal exposure of the second, affectively relevant, word. It was emphasised that participants did not have to memorize the words, nor did they have to search for a relation between the words. Finally, participants were asked to sit relatively close to the screen, at approximately 30 cm. After this, the first acquisition block was initiated. On each trial, a row of 10 X’s was first presented for 500 ms; this is 35 cycles on a 70-Hz screen. A neutral word immediately followed the X’s, remaining on the screen for 2500 ms (175 cycles). Then a positive or a negative word was presented for 28.571 ms (2 cycles), followed by a row of 10 X’s presented for 500 ms (35 cycles). Whereas the first row of X’s served only as a warning signal for the participants, the second row of X’s was more important because it served as a backward mask for the briefly presented ASi. The backward mask used in this study is likely to be very effective because (a) it is highly similar to the ASi in that both the mask and the ASi consist of letters, and because (b) the interval between the offset of an AS and the onset of the mask is very small (virtually 0). Both factors have proven to be important determinants of the effective¨ hman, 1993). The detectability of the ness of a mask (Holender, 1986; Esteves & O AS is even more reduced by the presentation of the NS just before the AS, because as such the NS functions as a forward mask. The intertrial interval was 3 s. After the first acquisition block was completed, a ready message appeared on the screen. After approximately 15 s the second acquisition block was started. In each of the two acquisition blocks, all 24 neutral words were presented once. The order of presentation of neutral, negative, and positive stimuli was randomized before each block, of course with the restriction that each neutral word had to be followed by an AS of the same valence in both blocks. Due to the randomization, however, a neutral word was not necessarily followed by the same AS in both blocks. After the second acquisition block, the ready message reappeared. During the test block which followed, all neutral words were presented again, this time not followed by an AS. Participants were instructed to indicate whether the presented word made them feel positive or negative. They could do this by saying a number between 210 (very negative) and 110 (very positive). Awareness assessments. As in the study of De Houwer et al. (1994), after the NS– AS pairs were presented and after NSi were evaluated, participants were asked (a) whether they had noticed anything strange and (b) whether they had noticed the presence of a second word immediately after the clearly visible neutral word. Participants who said to have seen a briefly presented affective word were regarded as aware. Afterwards, participants were informed that all NSi were immediately followed by a very briefly presented positive or negative word. They were then told that all stimuli would be presented again and were asked to direct all their attention to the word which immediately followed the neutral word. On each trial, they had to decide whether the briefly presented word had a positive or a negative meaning. The sequence of events on each of the 24 discrimination trials was exactly the same
IMPLICIT EVALUATIVE LEARNING
as that on learning trials. Participants who (a) could report the identity of one or more of the ASi or (b) performed above chance on the discrimination task were regarded as aware of the presentations. It is important to note that during the discrimination block, each NS was followed by an AS of the same valence as the ASi that followed the NS during the two learning blocks. This provided us with a more direct way to assess whether participants have conscious knowledge of the presented NS-AS pairs. If participants would have, even partial, conscious knowledge of the NS–AS pairs, they could use this knowledge during the discrimination task to predict the valence of the briefly presented AS based on the NS that preceded it (Shanks & St. John, 1994). If discrimination performance is not different from chance, it would strongly suggest that participants have no conscious knowledge of the NS–AS pairs.4 Design. Two variables were manipulated: Condition and List. In the first condition, during learning trials, neutral words of the first list of 12 words were followed by briefly presented positive words, whereas neutral words of the second list were followed by briefly presented negative words. In the second condition, neutral words of the second list were followed by positive words, whereas neutral words of the first list were followed by negative words. Participants were alternatingly assigned to one of these conditions. An alternative to the present design would have been to randomly assign half of the NSi to positive ASi and the other half to negative ASi for each participant individually. However, using fixed lists has the advantage that error variability due to interstimulus differences in evaluation will be captured by the variable List. It was predicted that the words of List 1 would receive a higher evaluative rating in Condition 1 than in Condition 2, whereas the mean evaluative rating of the words of List 2 would be highest in Condition 2. This pattern would result in an interaction between Condition and List. Results Awareness. One participant reported the identity of three ASi when asked whether she noticed anything strange during the experiment. Two other participants identified two ASi during the discrimination block. Four participants performed above chance on the discrimination task, with above chance defined as 16 or more correct discriminations of 24 trials. According to a binomial distribution with the probability of a correct discrimination set at .50, the probability of making 16 or more correct discriminations is smaller than .08. These seven participants were regarded as aware of the ASi presentations. Evaluation of NSi. The data of the 50 unaware participants (24 in Condition 1 and 26 in Condition 2) were analyzed by means of a 2 (Condition) 3 2 (List) ANOVA.
Some authors would argue that a difference in discrimination performance between learned NS–AS pairs and unlearned NS–AS pairs can be regarded as evidence for conscious knowledge of NS–AS pairs (e.g., Shanks & St. John, 1994). In our opinion, such a difference could equally well be caused by unconscious knowledge of the pairs (Baeyens et al., 1994). However, we wanted to apply the most strict criteria possible, in order to avoid possible critical comments.
DE HOUWER, HENDRICKX, AND BAEYENS
TABLE 1 Mean Ratings of the Words of List 1 and List 2 in Condition 1 and Condition 2 of Experiment 1 (SD in Parentheses)
List 1 List 2
2.67 (1.48) 2.22 (1.83)
1.95 (1.38) 1.88 (1.52)
Table 1 shows the relevant means. The expected interaction between Condition and List failed to reach significance [F(1, 48) 5 1.11; MSe 5 0.79; n.s.]. NSi of the first list tended to be rated as more positive in the first condition than in the second condition. However, contrary to predictions, the NSi of the second list also tended to be rated as more positive in Condition 1 than in Condition 2. Neither the main effect of Condition [F(1, 48) 5 1.75; MSe 5 4.06; n.s.], nor the main effect of List [F(1, 48) 5 2.16; MSe 5 0.79; n.s.] reached significance. An additional item analysis was conducted by calculating for each NS, over participants, the mean evaluation when it was paired with a positive AS and when it was paired with a negative AS. Only the data of the unaware participants were taken into account. A comparison of these means revealed no significant effect of the AS valence [F(1, 23) , 1]. In order to explore whether the failure to observe a significant evaluative learning effect was due to excluding participants who were classified as aware, a second subject analysis was conducted which did include the data of the aware participants. Again, however, the expected interaction between Condition and List did not reach significance [F(1, 55) , 1]. Discrimination performance. Because the participant who identified ASi during the learning phase could already be considered as aware according to the subjective criterion, she was not given a subsequent valence-discrimination task. The mean number of correct discriminations of the remaining aware and unaware participants was 12.10 (SD 5 2.23), which did not differ from chance [t(55) , 1]. Discussion The present experiment was an exact replication of a previous study (De Houwer et al., 1994) in which a significant evaluative learning effect was found using subliminally presented ASi. In fact, in both studies the same computer program was used for presenting the stimuli during the learning and testing phase. Apart from possible subtle differences in lighting conditions and laboratory environment, the only important difference between both studies was that participants had to perform an additional valence-discrimination task, but this occurred only after the learning and testing phase. Nevertheless, the significant evaluative learning effect that was observed in the original study (De Houwer et al., 1994) could not be replicated. The evaluative learning effect was not significant in the present study, neither when only the data of the unaware participants were analyzed, nor when the data of the participants who were classified as aware were included in the analysis. In order to compare the results of both experiments statistically, we conducted an Experiment 3 Condition 3 List
IMPLICIT EVALUATIVE LEARNING
ANOVA, only taking into account the data of unaware participants. Importantly, the interaction among the three variables was not significant [F(1, 108) , 1], nor were any other effects that involved the variable Experiment [all Fs , 1.72]. The interaction between Condition and List, however, was significant [F(1, 108) 5 4.60; MSe 5 .92]. These results suggest that the original and present studies were statistically homogeneous and that the absence of a significant evaluative learning effect in the present study might have been due to sampling variability. On the other hand, it cannot be excluded that the originally observed effect was not genuine but rather due to chance. In order to further investigate the reliability of evaluative learning with subliminally presented ASi, we decided to run a number of additional experiments. Before we describe these experiments, it needs to noted that performance on the discrimination task in Experiment 1 showed that participants could not discriminate positive from negative ASi despite the fact that they were asked to direct their attention to the ASi. It can therefore be concluded that overall, ASi were presented too briefly to allow for above chance discrimination performance. It could be argued that this conclusion also holds for our primary study (De Houwer et al. 1994), because that experiment was (apart from the discrimination task) identical to the present one. The same logic was used by Krosnick et al. (1992) when, based on the results of a pilot study, they argued that in their actual experiment AS valence could not be detected. Because significant effects were obtained in our primary study, one could conclude that evaluative learning can occur even if the valence of the ASi can not be detected. However, more data are needed in order to substantiate this claim. EXPERIMENT 2
There were some small procedural differences between Experiments 1 and 2. In Experiment 1, 12 different positive and 12 negative words were used as ASi. In Experiment 2, only 1 positive and 1 negative word was used, being the word with the most extreme positive or negative evaluation. We hoped that this would increase the magnitude of the evaluative learning effect. A second difference was that instead of existing neutral words, nonexisting nonsense words were used as NSi in Experiment 2. Evidence from conditioning studies indicates that previously unknown stimuli more readily enter into new associations (Cacioppo, Marshall-Goodell, Tassinary, & Petty, 1992). A final difference was that only the subjective awareness measure was used. As was argued before, using this measure is a very conservative way of verifying whether NS–AS associations were consciously detected. It was decided to delay the (even more conservative) combined use of the subjective and valence-discrimination measure to a further experiment that would be conducted if significant effects were found here. Method Research participants. Fifty-three first-year psychology students participated in order to partially fulfil course requirements. None had participated in the previous experiment. Materials. The most extreme positive word and the most extreme negative word were selected from the words used in the pilot study of De Houwer et al. (1994).
DE HOUWER, HENDRICKX, AND BAEYENS
The positive word was ‘‘geluk’’ (happiness) which had a mean rating of 5.77 on a 210 to 110 scale, the negative word was ‘‘kanker’’ (cancer) which had a mean rating of 25.33. The 24 nonsense words listed in Appendix 2 were used as NSi. They were selected from the materials used by Solomon and Postman (1952) and Yazuv (1963) and have no meaning in the Dutch language. Procedure. Participants were told that they would see nonexisting, meaningless words. It was said that the only thing they had to do was to look attentively at the words. They should, for instance, not try to remember them. For each participant, the computer randomly assigned half of the nonsense words to the positive AS and the other half to the negative AS. Thus, in contrast to Experiment 1, the set of words that were paired with a positive AS or a negative AS differed from participant to participant. Compared to using fixed lists (as in Experiment 1), random assignment increases the generalizability of the results. We did not anticipate a dramatic increase in error variability because we expected that there would be only small overall differences in the initial liking of the various nonsense words. All other aspects of the procedure were identical to those of Experiment 1. Awareness measure. It was only assessed whether participants subjectively experienced the presentation of the ASi. This was done by asking the same questions as in Experiment 1. Also, the criteria for identifying aware participants were identical. Results and Discussion Only one participant identified one of the ASi during the learning phase. An ANOVA performed on the data of the 52 unaware participants showed that nonsense words that were paired with the positive AS were evaluated as more positive (M 5 1.12; SD 5 1.43) than nonsense words that were paired with the negative AS (M 5 0.61; SD 5 1.65) [F(1, 51) 5 4.97; MSe 5 1.39; p , .05]. An item analysis showed that words were evaluated as more positive when paired with a positive AS than with a negative AS [F(1, 23) 5 5.34; MSe 5 .85; p , .05]. Means were based on the data of the unaware participants. Contrary to Experiment 1, we now obtained a statistically significant evaluative learning effect with subliminally presented stimuli. This result basically replicates the findings of our primary study (De Houwer et al., 1994). Although the evaluative learning effect was statistically significant, overall it was small and still showed a large variability. We therefore decided to run a replication experiment in which we tried to further optimize certain procedural aspects. EXPERIMENT 3
Experiments 2 and 3 mainly differed on three points. First, NS–AS pairs were presented four times instead of two times. In studies with supraliminally presented stimuli, it has been shown that evaluative learning is enhanced by increasing the number of NS–AS pairings up to a certain level (Baeyens et al., 1992). It could therefore be expected that increasing the number of NS–AS pairings would also increase the magnitude of the evaluative learning effect when ASi are presented subliminally. Second, instead of randomly assigning the nonsense words to either a positive or a negative word for each participant individually, fixed lists were used in
IMPLICIT EVALUATIVE LEARNING
Experiment 3 (as was the case in Experiment 1). We returned to the original design because, contrary to our expectations, substantial individual differences in the liking of the various nonsense words were observed in Experiment 2. As was mentioned before, with random assignment, interstimulus baseline differences increase error variability, whereas with fixed lists, this variability is captured by the variable List. Given the successful results of Experiment 2, it was decided to assess both subjective awareness of the ASi and valence-discrimination performance. By using two alternative awareness measures simultaneously, the hypothesis of implicit evaluative learning could be tested in an even more conservative way than in Experiment 2. The valence-discrimination task also provided the opportunity of investigating whether the low levels of discrimination performance observed in Experiment 1 could be replicated. Finally, if significant evaluative learning was found in participants who could not discriminate above chance, this would directly demonstrate that evaluative associations can be learned even if the valence of the ASi cannot be detected. Method Research participants. Fifty-four first-year psychology students participated in order to partially fulfil course requirements. None had participated in Experiments 1 or 2. Materials. The same NSi and ASi were used as in Experiment 2. The only difference was that the 24 nonsense words (NSi) were divided into two lists of 12 words. Based on the mean ratings for the individual nonsense words obtained in Experiment 2, the lists were constructed in such a way that the mean rating of the words in List 1 (M 5 .76; SD 5 2.02) and List 2 (M 5 .89; SD 5 1.52) did not differ significantly [F(1, 22) , 1]. Although the ratings of the nonsense words obtained in Experiment 2 were influenced by the experimental manipulations in that experiment, it can still be assumed that they provide some indication of the overall differences in liking between the individual nonsense words. The composition of the lists can be found in Appendix 2. Procedure. As in Experiment 1, participants were alternatingly assigned to one of two conditions. In Condition 1, the nonsense words of List 1 were paired with the positive AS and the words of List 2 with the negative AS, whereas the opposite was true for the participants in Condition 2. All NS-AS pairs were presented four times, one time in each of four learning blocks. The first and second, and third and fourth, blocks were separated by a brief pause of a few seconds. Between the second and third learning blocks, however, there was delay of approximately 20 min during which participants participated in an unrelated experiment. This delay was introduced to avoid participants becoming disinterested in the presented stimuli and developing a negative attitude toward the experiment, which could interfere with evaluative learning. The sequence of events on a trial was the same as in Experiments 1 and 2, expect that the first row of X’s was presented for 1000 instead of 500 ms. The order of presentation within each block was random, with the restriction that no more than three words of the same list were presented in a row. The testing phase and the awareness questions were identical to those of previous experiments. After the awareness questions had been answered, participants were informed about the presentation
DE HOUWER, HENDRICKX, AND BAEYENS
TABLE 2 Mean Ratings of the Nonsense Words of List 1 and List 2 in Condition 1 and Condition 2 of Experiment 3 (SD in Parentheses)
List 1 List 2
1.50 (2.02) 1.69 (1.75)
0.62 (1.05) 1.75 (1.30)
of the ASi and were given 24 discrimination trials. The sequence of events on the discrimination trials was the same as that on learning trials. Instructions for the discrimination task were the same as in Experiment 1. Contrary to Experiment 1, during the discrimination trials, NSi were not necessarily followed by an AS of the same valence as during the learning trials. Instead, half of the words of each list were followed by the positive AS and half were followed by the negative AS. As a consequence, only half of all nonsense words were followed by the same AS as during the learning trials. This provided us with a more direct measure of participants’ conscious knowledge of the NS–AS pairs. As was mentioned before, if participants had even partial conscious knowledge of the NS–AS pairs, they could use this knowledge during the discrimination task by predicting the valence or even the identity of the briefly presented AS based on the NS that preceded it (Shanks & St. John, 1994). Therefore, discrimination performance should be better on trials where the NS was followed by the same AS as during learning. If no such difference is observed, it would strongly suggest that participants have no conscious knowledge of the NS– AS pairs. Results Awareness. None of the participants identified an AS during the learning phase. During the discrimination task, three participants identified an AS and one participant performed above chance (17 correct discriminations on 24 trials). These four participants were regarded as being aware of the presentation of the ASi. Evaluation of NSi. The data of the unaware participants (24 in Condition 1, 26 in Condition 2) were analyzed by means of a 2 (Condition) 3 2 (List) ANOVA. Table 2 gives the relevant means. The expected interaction between Condition and List reached significance [F(1, 48) 5 4.41; MSe 5 1.26; p , .05]. Means show that the nonsense words of List 1 were evaluated as more positive in Condition 1 than in Condition 2, whereas the words of List 2 were evaluated as more positive in Condition 2 than in Condition 1, but only the first difference reached significance on a post hoc Tukey HSD test [p , .01]. Unexpectedly, the main effect of List was also significant [F(1, 48) 5 8.59; MSe 5 1.26; p , .01], showing that the nonsense words of the second list were overall evaluated as more positive than the words of the first list. The main effect of Condition was not significant [F(1, 48) 5 1.15; MSe 5 3.69; n.s.]. An item analysis performed on the data of the unaware participants showed a marginally significant effect of the type of AS that was paired with a NS [F(1, 23) 5 3.00; MSe 5 0.85; p , .10]. As expected, a nonsense word tended to be evaluated
IMPLICIT EVALUATIVE LEARNING
as more positive when it was previously paired with a positive AS than when it was paired with a negative AS. Discrimination performance. Aware and unaware participants made a mean number of 11.46 (SD 5 1.96) correct discriminations on 24 trials, which does not differ from chance [t(53) , 1]. Also, discrimination performance was not different on trials where a NS was followed by the same AS as during learning (M 5 5.80; SD 5 1.75) than on trials where a different AS followed (M 5 5.67; SD 5 1.78) [F(1, 53) , 1]. Discussion The present experiment replicated and extended the results of Experiment 2 and our primary experiment (De Houwer et al., 1994). Significant evaluative learning was observed in participants who (a) did not subjectively experience the presentation of the ASi and (b) could not even reliably determine whether a positive or negative AS was presented when they were asked to direct their attention to the briefly presented ASi. Overall, discrimination performance was not different from chance, both on trials where a NS was followed by the same AS as during learning and on trials where a different AS followed. The latter result strongly suggests that participants had no conscious knowledge of the presented NS–AS pairs. Because ASi were presented in exactly the same way as in Experiment 2, the present discrimination data suggest that in Experiment 2, participants also could not detect the valence of the ASi. In conclusion, the results clearly support the hypothesis that evaluative associations can be learned implicitly and can even be learned if the valence of the ASi cannot be detected. On the other hand, the observed evaluative learning effect was once again weak and this time only statistically reliable in the subject analysis, despite the fact that various aspects of the procedure of Experiment 2 were optimized in this experiment. For instance, NS–AS pairs were presented four instead of two times, which would normally increase the effect (Baeyens et al., 1992). However, it is possible that the rather long delay between the first two and the last two pairings prevented incremental learning, so that only the last two pairings affected the evaluative ratings. Another possibility, of course, is that with subliminally presented ASi, evaluative learning does not increase with an increase in the number of NS–AS pairings. In order to investigate this issue more directly, we conducted a fourth experiment in which NS– AS pairs were presented either two or four times. A second reason for conducting this experiment was that after the successful results of Experiment 2 and 3, we felt that the time had come to start examining the properties of evaluative learning with subliminally presented ASi. Manipulating the number of pairings was an obvious first candidate because, as we just suggested, it might reveal a difference between evaluative learning with detectable and not detectable ASi. EXPERIMENT 4
Method Research participants. Thirty-three first-year psychology students participated in order to partially fulfil course requirements. None had participated in an earlier evaluative learning study with subliminally presented ASi.
DE HOUWER, HENDRICKX, AND BAEYENS
Materials. Twelve of the nonsense words used in Experiments 2 and 3 were used as NSi. These 12 words were divided into four lists of 3 words each. Based on the evaluative ratings of Experiment 3, the lists were constructed in such a way that they did not differ in the mean evaluation of the words [F(1, 4) , 1, for all comparisons]. As in Experiments 2 and 3, cancer was used as the negative AS and happiness as the positive AS. Procedure. The nonsense words of a particular list could be assigned to either the positive or the negative AS and could be paired two or four times. As such, there were four possible presentation arrangements. For each participant, one list was assigned to each possible arrangement. Over participants, lists were rotated over the four possible arrangements. Because six NS–AS pairs were presented two times, and six pairs four times, the total number of trials was 36. The order of the trials was randomized for each participant individually with the restriction that the same AS could not be presented on more that three consecutive trials. After the 18th trial there was a brief pause of a few seconds. Each trial was structured in the same way as in Experiment 3. The testing phase was again the same as in previous experiments, and afterward, the two awareness questions used in previous experiments were asked. No subsequent discrimination trials were given. Design. The counterbalancing of lists over the four presentation arrangements was achieved by manipulating two between participants variables. The first variable determined which lists would be presented two or four times. If this variable (named List presentation assignment) was given the value one, Lists 1 and 2 would be presented four times, and Lists 3 and 4 two times. The opposite was true if the variable was given the value two. The second variable determined which lists would be assigned to the positive or the negative AS. If this variable (named List valence assignment) was given the value one, the words of Lists 2 and 4 were paired with the negative AS, whereas the words of Lists 1 and 3 were paired with the positive AS. With a value of two, the opposite was true. As such there were four between-participants conditions. Results None of the participants identified a word during the learning phase. Eight participants were run in three of the four conditions; in one condition (Lists 3 and 4 presented four times, Lists 1 and 3 paired with a negative US), nine participants were run. For each participant, four means were calculated: The mean evaluation of words paired with the positive AS and words paired with the negative AS were calculated, separately for the words paired two times and words paired four times. An ANOVA involving the two between participants counterbalancing variables (List presentation and List valence assignment) and two within-participants variables (AS valence and Number of Presentations) was conducted, using weighted means. We expected to find a main effect of AS valence, showing that words paired with a briefly presented positive AS would be evaluated as more positive than words paired with a negative AS. A second hypothesis was that the effect of AS valence would be larger with four presentations than with two presentations of NS–AS pairs. Neither prediction was confirmed. The main effect of AS valence did not reach
IMPLICIT EVALUATIVE LEARNING
significance [F(1, 29) , 1], nor did the interaction between AS valence and Number of Presentations [F(1, 29) 5 1.86; MSe 5 5.97; n.s.]. Whereas with four presentations, NSi that were paired with the positive AS tended to be rated as more positive (M 5 1.86; SD 5 3.20) than words paired with the negative AS (M 5 1.55; SD 5 3.31), the opposite was true for NSi that were only paired two times [.89 (SD 5 3.03) and 1.72 (SD 5 2.73) for NSi paired with the positive and negative AS, respectively]. Also, none of the other effects reached significance. For each nonsense word, the mean evaluation was calculated over participants when paired (a) two times with the positive AS, (b) two times with the negative AS, (c) four times with the positive AS, and (d) four times with the negative AS. An ANOVA, involving the variables AS valence and Number of Presentations was performed on these means. Neither the main effect of AS valence [F(1, 11) , 1], nor the interaction between AS valence and Number of Presentations [F(1, 11) 5 1.47; MSe 5 2.67; n.s.] reached significance. Discussion As in Experiment 1, we failed to obtain evidence for evaluative learning with subliminally presented ASi. With two presentations of NS–AS pairs, there even was a tendency for a reversed effect, with overall lower ratings for NSi paired with a positive AS. Because stimuli were presented in the same way as in previous experiments, we see no obvious reason for our failure to replicate. Error variability was extremely high. This is probably due to large interstimulus and interindividual differences in the liking of the various nonsense words. META-ANALYSIS
Until now, we have conducted five experiments with subliminally presented ASi, four reported in the present paper and one reported by De Houwer et al. (1994). The experiments involved a total number of 260 participants. In three of the experiments, significant evaluative learning effects have been obtained when analyzing the data of participants who were unaware of the presentation of the ASi according to either a subjective (noticing of the ASi) or objective (above chance discrimination of the affective connotation of the ASi) criterion. However, two experiments rendered nonsignificant results. In one experiment, a tendency toward a reversed learning effect was observed. Because different results were obtained in the different experiments, we wanted to examine whether the obtained set of results could have arisen from chance or whether, as a whole, the experiments provide statistically reliable evidence for evaluative learning with subliminally presented ASi. Meta-analytic techniques provide the accurate means for answering questions like these. Indeed, one of the primary goals of meta-analysis is to statistically summarize results from several studies so as to guide conclusions regarding the reliability of the examined effect (Strube & Hartman, 1983; Rosenthal, 1991). Moreover, meta-analysis also renders a cross-experimental estimate of the size of the effect and can be used to identify procedural elements which might influence the effect size (Rosenthal, 1991). r-Values were calculated as an estimate of effect size (Rosenthal, 1991) for each
DE HOUWER, HENDRICKX, AND BAEYENS
of the five experiments conducted at our lab, only taking into account the data of unaware participants. Based on the r-values, a meta-analysis was conducted which gave evidence for a statistically reliable [Z 5 3.25; p , .001] but small (weighted mean effect size r 5 .21) evaluative learning effect.5 Importantly, it was also found that the five experiments can be regarded as homogeneous [χ2(4) 5 4.05; n.s.]. This suggest that our failures to find significant evaluative learning effects in Experiments 1 and 4 should be mainly attributed to random sampling errors rather than to procedural differences between experiments. Figure 1 gives the distribution of the size of the individual evaluative learning effects shown by the 247 unaware participants. The individual evaluative learning effects were calculated by deducting the mean evaluation of the NSi paired with a negative AS from the mean evaluation of the NSi paired with a positive AS and, when necessary, adjusting for baseline differences in the evaluations of lists. The distribution of evaluative learning effects does not differ from a normal distribution with a mean of .32 and a standard deviation of 1.50 [χ 2(5) 5 7.24; n.s.]. The fact that the distribution is normal also suggests that one can not attribute the overall learning effect to the results of a limited number of participants who showed a large evaluative learning effect and who were actually aware of some AS presentations or NS–AS associations, but were not detected as being aware. Figure 1 also gives the distribution of the evaluative learning effect shown by participants who did detect ASi or whose discrimination performance was above chance. It is interesting to note that, overall, these participants tended to show a reversed evaluative learning effect (M 5 20.37; SD 5 2.15). A reversed effect was mainly present in participants who discriminated above chance (M 5 21.05; SD 5 2.68 ), but also participants who noticed the presentation of ASi showed a smaller effect than unaware participants (M 5 0.18; SD 5 1.79). Of course, this observation must not be given too much weight because of the small number of data points on which it is based. However, it suggests that even if some of the participants classified as unaware were actually aware of the presentation of the ASi, these undetected aware participants could not be responsible for the observed evaluative learning effect. Congruent with this line of reasoning, it was observed that the distribution of evaluative learning effects shown by both aware and unaware participants did not differ from a normal distribution, with a mean of 0.29 and a standard deviation of 1.54. GENERAL DISCUSSION
The present research aimed to examine whether associative evaluative learning depends upon the conscious identification of the relevant NS–AS associations. Our approach consisted of pairing NSi with ASi that were presented too briefly to allow for conscious identification. This approach is a very conservative one: It is not just assessed whether participants have consciously detected NS–AS associations, it is 5
This meta-analysis does not take into account the data of two other studies with subliminally presented ASi: A pilot study (N 5 19) conducted at our laboratory and a sixth experiment (N 5 80) that was independently conducted by a colleague (Jos Feys, nonpublished data) at the University of Leuven. When the results of these studies (only unaware participants) are included in the meta-analysis, similar effects are obtained [Z 5 3.37; p , .0005; weighted r 5 .18].
IMPLICIT EVALUATIVE LEARNING
FIG. 1. Distribution of evaluative learning effects of unaware participants, participants who noticed ASi, and participants who discriminated above chance in Experiments 1-4 and in De Houwer et al. (1994).
assessed whether a necessary (but not sufficient) precondition for conscious identification of associations is met, namely, whether both elements are consciously perceived. Despite the conservative test, our main hypotheses was confirmed: Statistically reliable evaluative learning effects were obtained even though (a) participants did not subjectively experience the presentation of the ASi and (b) ASi were presented too briefly to allow for reliable discrimination of their affective valence. Although effects were not always significant, a meta-analysis confirmed that the present procedure results in statistically reliable evaluative learning with subliminally presented stimuli. As such, our data give further support for the hypothesis that evaluative learning does not depend upon conscious knowledge of the relevant NS–AS associations.
DE HOUWER, HENDRICKX, AND BAEYENS
A second aim of the present experiments was to investigate whether evaluative learning can be observed when the valence of the ASi can not be detected. Results also confirmed this hypothesis. In both Experiments 1 and 3, participants’ mean discrimination performance was—as a group—not different from chance. This shows that with the employed masking stimuli and presentation times, the valence of the ASi cannot be detected. Nevertheless, in three studies with (almost) identical presentation procedures, significant evaluative learning effects were obtained. The most direct evidence was obtained in Experiment 3, where significant evaluative learning was found in the very same participants who could not discriminate above chance. Some might still argue that our discrimination test was not powerful enough to detect reliable discrimination of valence because it consisted of a limited set of trials. However, the number of trials used is clearly acceptable (Cheesman & Merikle, 1984). Moreover, as we have stressed before, during the discrimination task, conditions were far more optimal for conscious perception to occur than during the actual learning trials. Unlike during learning, participants knew during discrimination that ASi would be briefly presented and were asked to direct all their attention to the ASi. Also, the discrimination trials were given after the learning trials. Because perception becomes more sensitive during the course of an experiment (Cheesman & Merikle, 1984; Holender, 1986), this should also increase the power of the discrimination test (Eriksen, 1980). Therefore, our discrimination test can be regarded as conservative and powerful by most standards (Eriksen, 1980; Cheesman & Merikle, 1984). It must be reemphasised that our two main conclusions do not depend upon whether the presentation of the ASi was ‘‘truly’’ subliminal. It is for instance possible that with the present presentation parameters, participants could perform above chance on some discrimination tasks (e.g., presence–absence decisions) even though they cannot detect the valence of the briefly presented stimuli. Some might interpret such a finding as showing that the ASi were not presented ‘‘truly’’ subliminal. We need not enter this debate given the aims of the present research. In order to establish that the observed evaluative learning effects did not depend upon conscious knowledge of the NS–AS pairs, it was examined whether the ASi were subjectively experienced. Because they were not, it seems logical to conclude that the observed evaluative learning was implicit, regardless of whether or not the ASi were presented ‘‘subliminally’’ according to certain more ‘‘objective’’ criteria. The fact that participants could also not discriminate between positive and negative ASi when asked to do so gives further support to the hypothesis that NS–AS associations could not have been consciously detected and additionally shows that evaluative learning can be observed even when participants can not discriminate the valence of the ASi. Although significant evaluative learning effects have been found in three of the five experiments conducted at our laboratory, two experiments rendered nonsignificant results. As was confirmed by the results of the meta-analysis, the pattern of results suggests that the procedure used in our experiments induces only small and variable effects that will not always reach significance. Other procedures which have been used to show evaluative learning with subliminally presented stimuli (Krosnick et al., 1992; Niedenthal, 1990) also resulted in only small effects. Of course, this complicates future efforts to investigate the actual processes involved in evaluative
IMPLICIT EVALUATIVE LEARNING
learning with subliminally presented stimuli. Progress in this area will depend upon the development of alternative, more powerful procedures. One can try to optimize the procedure used in the present experiments. For instance, error variability due to interstimulus and interindividual differences in liking of the various NSi could be substantially reduced by asking participants to evaluate the NSi both before and after the learning phase. Despite the fact the until now only small evaluative learning effects have been reported when ASi are presented subliminally, the effect appears to be genuine. If genuine, it has important theoretical implications. Because the effect shows that evaluative learning can be implicit, it supports the hypothesis that evaluative learning is qualitatively different from other forms of associative learning (Baeyens & De Houwer, 1995) such as classical conditioning which does depend upon conscious knowledge of the associations (e.g., Dawson & Shell, 1987). The present results are also relevant for research on implicit learning in general (De Houwer et al., submitted for publication). In implicit learning research, the central question has been whether learning can occur without awareness of the information that is being learned (Shanks & St. John, 1994). Commonly, awareness measures are used to check whether or not the knowledge acquired during learning is available to consciousness. Some authors dismiss existing evidence for implicit learning based on the argument that in many studies, the awareness measure was either not sensitive enough or did not look for all useful conscious knowledge participants could possess (e.g., Perruchet & Pacteau, 1990; Shanks & St. John, 1994). Studies with subliminally presented stimuli are particularly relevant because they can not easily be criticized based on the awareness measure. As was argued before, the awareness measure in these studies is very sensitive and very conservative: It is not just assessed whether the associations themselves have been consciously detected, but it is assessed whether an essential precondition for conscious identification of associations has been met, namely, the subjective experience of perceiving the elements of the association.
APPENDIX 1 Neutral Words of Lists 1 and 2 and Negative and Positive Words Used in Experiment 1 Neutral List 1 lijn (line) balpen (ball pen) emmer (bucket) cijfer (numeral) term (term) duim (thumb) ingang (entrance) deur (door) bus (bus) kip (chicken) voetpad (footpath) hek (fence)
Neutral List 2
raam (window) hendel (handle) lade (drawer) kist (box) staan (to stand) melk (milk) steen (stone) scherm (screen) baard (beard) jas (coat) bord (board) tas (cup)
kanker (cancer) oorlog (war) eenzaam (lonely) haat (hate) moord (murder) falen (to fail) dood (death) verdriet (sadness) pijn (pain) slecht (bad) vrees (fear) geweer (gun)
geluk (happiness) zacht (soft) eerlijk (honest) lief (sweet) vrede (peace) teder (tender) moeder (mother) strelen (to caress) genieten (to enjoy) vreugde (joy) mooi (beautiful) samen (together)
DE HOUWER, HENDRICKX, AND BAEYENS
APPENDIX 2: Nonsense Words Used in Experiments 2–4 aslan (1, 4) enanwal (1, –) ilim (1, –) udibnon (1, –) enshimi (1, –) biwojni (1, 3) afworbu (1, –) yalan (1, –) zahmet (1, 2) zarif (1, 3) saricik (1, 4) nansoma (1, 4)
kadirga (2, –) zabulon (2, –) lokanta (2, –) civadra (2, 3) mecburi (2, 1) bayram (2, –) ortak (2, 2) merkez (2, 1) nijaron (2, 2) fevkani (2, –) tavhane (2, –) borulce (2, 1)
Note. The numbers of the lists to which the words were assigned in Experiment 3 (first number) and Experiment 4 (second number) are shown in parentheses.
ACKNOWLEDGMENTS Jan De Houwer is a research assistant and Frank Baeyens a postdoctoral researcher for the National Fund for Scientific Research (Belgium). The authors gratefully acknowledge the help of Noe¨l Bovens in preparing the experiments, Geert Crombez in performing the meta-analysis, and Paul Eelen for his helpful comments on an earlier draft.
REFERENCES Baeyens, F., Crombez, G., Van den Bergh, O., & Eelen, P. (1988). Once in contact, always in contact: Evaluative conditioning is resistant to extinction. Advances in Behaviour Research and Therapy, 10, 179–199. Baeyens, F., & De Houwer, J. (1995). Evaluative conditioning is a qualitatively distinct form of classical conditioning: A reply to Davey (1994). Behaviour Research and Therapy, 33, 825–831. Baeyens, F., De Houwer, J., & Eelen, P. (1994). Awareness inflated, evaluative conditioning underestimated. Behavioral and Brain Sciences, 17, 396–397. Baeyens, F., Eelen, P., Crombez, G., & Van den Bergh, O. (1992). Human evaluative conditioning: Acquisition trials, presentation schedule, evaluative style, and contingency awareness. Behaviour Research and Therapy, 30, 133–142. Baeyens, F., Eelen, P. & Van den Bergh, O. (1990b). Contingency awareness in evaluative conditioning: A case for unaware affective-evaluative learning. Cognition and Emotion, 4, 3–18. Baeyens, F., Eelen, P., Van den Bergh, O., & Crombez, G. (1989). The influence of CS-US perceptual similarity/dissimilarity on human evaluative learning and signal learning. Learning and Motivation, 20, 322–333. Baeyens, F., Eelen, P., Van den Bergh, O., & Crombez, G. (1990a). Flavor-flavor and color-flavor conditioning in humans. Learning and Motivation, 21, 434–455. Baeyens, F., Hermans, D., & Eelen, P. (1993). The role of CS-US contingency in human evaluative conditioning. Behaviour Research and Therapy, 31, 731–737. Cacioppo, J. T., Marshall-Goodell, B. S., Tassinary, L. G., & Petty, R. E. (1992). Rudimentary determinants of attitudes: Classical conditioning is more effective when prior knowledge about the attitude stimulus is low than high. Journal of Experimental Social Psychology, 28, 207–233. Cheesman, J., & Merikle, P. M. (1984). Priming with and without awareness. Perception and Psychophysics, 36, 387–395.
IMPLICIT EVALUATIVE LEARNING
Dawson, M. E., & Shell, A. M. (1987). Human autonomic and skeletal classical conditioning: The role of conscious factors. In G. Davey (Ed.), Cognitive processes and Pavlovian conditioning in humans (pp. 27–57). New York: Wiley. De Houwer, J., Baeyens, F., & Eelen, P. (1994). Verbal evaluative conditioning with undetected US presentations. Behaviour Research and Therapy, 32, 629–633. De Houwer, J., Baeyens, F., & Hendrickx, H. (submitted for publication). Implicit learning of evaluative associations. Eriksen, C. W. (1980). The use of a visual mask may seriously confound your experiment. Perception and Psychophysics, 28, 89–92. ¨ hman, A. (1993). Masking the face: Recognition of emotional facial expressions as a Esteves, F., & O function of the parameters of backward masking. Scandinavian Journal of Psychology, 34, 1–18. Hammerl, M., & Grabitz, H.-J. (1993). Human evaluative conditioning: Order of stimulus presentation. Integrative Physiological and Behavioral Science, 28, 191–194. Holender, D. (1986). Semantic activation without conscious identification in dichotic listening, parafoveal vision, and visual masking: A survey and appraisal. Behavioral and Brain Sciences, 9, 1–66. Jaanus, H., Defares, P. B., Zwaan, E. J. (1990). Verbal classical conditioning of evaluative responses. Advances in Behavioral Research and Therapy, 12, 123–151. Krosnick, J. A., Betz, A. L., Jussim, L. J., & Lynn, A. R. (1992). Subliminal conditioning of attitudes. Personality and Social Psychology Bulletin, 18, 152–162. Levey, A. B., & Martin, I. (1975). Classical conditioning of human ‘‘evaluative’’ responses. Behaviour Research and Therapy, 13, 221–226. Merikle, P. M., & Cheesman, J. (1986). Consciousness is a ‘‘subjective’’ state. Behavioral and Brain Sciences, 9, 42. Niedenthal, P. M. (1989). Implicit perception of affective information. Journal of Experimental Social Psychology, 26, 505–527. Perruchet, P., & Pacteau, C. (1990). Synthetical grammar learning: Implicit rule abstraction or explicit fragmentary knowledge? Journal of Experimental Psychology: General, 119, 264–275. Rosenthal, R. (1991). Meta-analytic procedures for social research. London: Sage. Shanks, D. R., & St. John (1994). Characteristics of dissociable human learning systems. Behavioral and Brain Sciences, 17, 367–395. Solomon, P. W., & Postman, L. (1952). Frequency of usage as determinant of recognition thresholds for words. Journal of Experimental Psychology, 43, 195–201. Staats, C. K., & Staats, A. W. (1957). Meaning established by classical conditioning. Journal of Experimental Psychology, 54, 74–80. Strube, M. J., & Hartmann, D. P. (1983). Meta-analysis: Techniques, applications, and functions. Journal of Consulting and Clinical Psychology, 51, 14–27. Stuart, E. W., Shimp, T. A., & Engle, R. W. (1987). Classical conditioning of consumer attitudes: Four experiments in an advertising context. Journal of Consumer Research, 14, 334–349. Yazuv, H. S. (1963). The retention of incidentally learned connotative responses. The Journal of Psychology, 55, 409–418. Zelner, D. A., Rozin, P., Aron, M., & Kulish, C. (1983). Conditioned enhancement of human’s liking for flavor by pairing with sweetness. Learning and Motivation, 14, 338–350. Received May 7, 1996