Evaluation methods for improving CAL materials

Evaluation methods for improving CAL materials

EVALUATION METHODS FOR IMPROVING CAL MATERIALS T. SAKAMOTO’,K. KIMUKA~,M. SHIMADA~,S. OZAWA~ and T. OKAMOT~ 'TokyoInstitute of Technology, ‘Chemic...

830KB Sizes 2 Downloads 25 Views

EVALUATION

METHODS

FOR IMPROVING

CAL MATERIALS

T. SAKAMOTO’,K. KIMUKA~,M. SHIMADA~,S. OZAWA~ and T. OKAMOT~ 'TokyoInstitute of Technology, ‘Chemical Engineering High School, ‘Vocational Training College, %eio University and ‘Kanazawa Institute of Technology Abstract-The main objective of the study was to develop systematic methods for improving materials by comparing the five coursewares which were produced by independent authors and related to high school matbemati~s, science and engineering. Various kinds of evaluation were undertaken before. ;mmed~ateiy before, during and imm~iatefy after CAL implementation; e.g. evaluators’ check on the drafts for courseware; evaluators’ check on the problems encountered in each frame while studying at the terminals; the readiness, pre- and post-tests; an anxiety scale; questionnaires for assessing any difficulties with learning in each chapter and each course; the course test; an image scale. an attitude scale and a line-connecting type of content analysis. Each courseware was evaluated and compared with others on the basis of the data analysis of tests and questionnaires as well as the learning history records, e.g., total learning duration, total number of passed frames, frequency of use of function keys, actual transitive path graph, actual average branch ratio, relative entropy in each frame and the index of educationat effectiveness. A number of important evaluation &ems were selected by factor analysis. Finalty an evaiuation package instrument for improving CAL eourseware were developed. CAL

~MPROYE~E~T

OF

CAL

MATERIALS

The ~rnpr~ve~e~~ of educational

effectiveness by CAL requires three factors to be evaluated: hardware, software and courseware. Of these, the courseware evaluation is the most important from the instructional viewpoint. Various kinds of information needs to be gathered; whether the amount and difficulty of the content, the sequence, and the degree of computer interaction are appropriate for students; how compatible it is with students’ attitudes and competence; and how it can help them attain good performance. Given this information, we could improve courseware and make it more effective. The evaluation studies on the educational effectiveness of CAL could be conducted from various viewpoint [l], Though some studies were concerned with CAL courseware, they were mostly done from the sllmmative viewpoint. Of recent work, the PLATO IV evaluation study by ETS is most eminent [2]. However, the systematic study on CAL courseware from a formative viewpoint has been scarcely conducted. This kind of evaluation study is particularly important during the developing stages of CAL, because summative evaluation, if it yields poor results, will tend to inhibit further creative developments. For this reason, we have developed the systematic formative evaluation package instrument for improving CAL courseware. In 1977, the Japan Society for the Promotion of Machine Industry (JSMI) provided a research grant for a group of CAL research workers in six sub-groups, scattered throughout Japan. Five of these sub-groups were given the task of developing CAL courseware for science or technology courses in high schools, while the sixth provided evaluation. This article is concerned with that evaluation study. PURPOSE

OF THE

PROJECT

Our project had the following objectives: (a) To compare the courseware produ~d by the five ~nde~ndent subgroups; (b) As a result, to recommend specific improvements; and (c) In parallel, to develop systematic methods for improving CAL materials. In this report, particular emphasis will be placed on (c)

METHODS Tke CAL

FOR

EVALUATING

CAL

MATERIALS

systm

The system used for the project was the JSPMI PEARL system based on the HITAC 10 mini-computer, which has two kinds of terminal. The Type II terminal has a random access slide projector

282

T.

SAKAMOTC) rt ul.

and a keyboard, whilst Type I has, in addition, auditory device with headphones.

a CRT display with light pen and a random

access

Cllurseware The five courseware Programme

3: 4: 5:

Period

developed

were as follows:

1: ‘Automatic 2:

Programmes

modules

Control’ for 1st grade pupils in technical ‘Waves’ for 1st and 2nd grade pupils in ‘Electric Circuits’ for 1st and 2nd grade pupils in ‘Kirchoffs Laws’ for 1st grade pupils in technical ‘Probability’ for 1st and 2nd grade pupils in

3 and 5 were designed

high school. high school high school. high school. technical

high school

for Type I terminals,

the others for Type 11.

of eaaluation

The period of the evatuation was from f lth November 1976 to 10th March 1877. A totaf of 106 pupils was divided into 5 groups. Each group used one CAL pro~ramme over one or two days from 20th January to I6 February 1977. There were 11 evaluators. That is, one associated with each module and six functioning overall. Evaluation

proc&rre

The procedure was as follows. (1) Before CAL implementation:

(a) authors’ materials, including drafts for slides and text, flowcharts, items for readiness and pre-and post-tests for measuring achievement, behavioural objectives, etc., were checked; (b) tive evaluators examined the problems encountered in each frame when it was being used at the terminals. (2) Immediately

before CAL

exercises,

pupils

were asked:

(a) to complete the readiness test. and pre-test; (b) to complete a questionnaire on learning motivation their preconceived images of CAL (Appendix A). (3) Ruring

CAL exercises.

(a] a questionnaire

pupils were asked to respond

designed

to measure

{Appendix B]; and (b) another at the end of each chapter, learning (Appendix D). (4) Immediately

after the CAL exercises,

p~tpils

were

designed

state at 30 to 60 minutes from the start

on slides, for assessing

any difficulties

module was evaluated

with

asked to complete:

(a) a questionnaire presented on the display at the end of the course for assessing with learning (Appendix I?); (b) the course test: (c) an image scale. as in 2(b) (Appendix A); (d) a questionnaire for measuring attitude towards CAL (Appendix C); (e) content analysis of courseware (Fig. 2). (5) Finally, each courseware results of:

to gauge

to:

their anxiety presented

and a questionnaire

and compared

any dj~~uItjes

with the others on the basis of the

(a) the data analysis of learning history records; (b) the data analysis of tests and questionnaires. Then a systematic method for evaluating CAL courseware results. Finally the e~~~luation package instrument for improving

was developed, CAL courseware

derived

from the above

was constructed.

283

Evaluation methods for improving CAL materials INFORMATION Information

FOR

IMPROVING

CAL

COURSEWARE

that could be gained before CAL implementation

The main items of information

gathered

are as follows:

(1) The way in which media were used and the teaching functions

distributed within and between frames were assessed, as illustrated in Table 1. For example, this shows that only one percent of frames in Programme 2 have the function of ‘hint’, while 46% have their function in Programme 4; and that only 19% of frames have the ‘instruction’ function in Programme 2, whilst 100’~ have this function in Programme 4. By assigning these kinds of figures to each teaching function or media utilization in each courseware module, effective information could be obtained for improving it. (2) A count made of the number and variety of anticipated student responses could give us information to allow adaption to meet students’ individual differences. and switches for correct and incorrect answers could also (3) Frequency of use of accumulators be utilized for making CAL courseware compatible with the differences of individual pupils. For example in Programme 2, the frequency of use is 16, giving a rate of use of 0.5. But in Programme 3, there is less adaptability with a frequency of use of 13 and a lower rate of 0.41. of (4) The pattern of transitive path graphs could also be utilized to generate an understanding the characteristics of CAL materials. For example, Fig. 1 shows that Programme 2 is simpler than Programme 3. (5) The average branching rate and the linearity rate could be calculated on the basis of the transitive path graph from the mean of the number of branches in each frame and the number of frames with only one path. For example, the average branch rate was 2.2 for Programme 3 and 1.6 for Programme 2 while the linearity rates were 0.71 and 0.48 respectively. By these figures we could interpret that Programme 2 had a larger number of frames with a smaller number of branches, whilst Programme 3 had a smaller number of frames with less frequent but extensive branching. Information teachers

that could be gained during CAL Implementation

from

the evaluation

by prqfessional

(1) Some errors in each frame could be found out. For instance, the misprints, errors and distractive guides to wrong answers, adequateness and variety of KR, timing of presentation in audio-visual display. Table 1. Analysis of characteristics of frame Rate of frame utilization Program 2 Program 4 (%) (%)

Teaching function Information Presentation Behavior Evocation Diagnosis (Response demand) K

R

Explanation. Description Hint Summary Question Instruction Choice Key Board Construction Note Intellectual Correct Wrong K R Comment Affective K R

1 2 3 4 5 6 7 8 9 10 11 12

25 1 3 22 19 1 7 0 2 1 26 1 1 27 1 0

34 46 9 38 100 3 9

Graph, Picture Illustration Character Sign, Equation Auditory Key Board C R T Text

13 14 15 16 17 18 19 20

48 I 7 97 0

12

34 3 1 2

Media Content of Slide

Presentation

Media

2

!UO 63

284

T. SAKA~O~ Gi014

~~EzJh--lEhv~Gl201 GlOl5 Programme

CiOlG

ANSO

Gl202

GICOI

GiOii’

et al. ANSO

G&019

Gl018

Gi020

GiO22A

Gl021

GYTZ

Gl02 2

2

03004

CLRl4

0300

837X4

KRCI3

K3005

Q3005

CLRI4

03005

R 83005

--+F Progromme

3

Fig. 1.

(2) General information could also be gained on the CAL courseware. For instance, the repetition of the same pattern; the distinctness of chapters; the compatibility with individual differences of learners; the level of difficulty; the consideration about motivation for learning. ~nf~r~ti~n

that rould be gained after CAL ~m~le~entat~o~ concerning uc~lieve~ent and questionnaire

Information was gained both from the results of readiness, pre- and post-tests, and from responses to questionn~res by pupils as follows: (1) The attainment of behavioural objectives in each pupil was measured by the equation: the quotient of effectiveness =

(Post test) -_-..____._--____ (Pre-test score) x loo (Full mark in pre-test) - (Pre-test score)

and the rate of modification in each test item =p

(The number of pupils who made wrong answer in pre-test, but made right answer in post-test) _____~~. (The total number of pupils who made wrong answer in pre-test) x 100.

The For was (2) The

larger the value in the equation, the more the educationa effectiveness of the CAL materiafs. example, average point of emotiveness in Programme 1 was 599 and one in Programme 2 89.3. degree of image change is given by the equation: Al,, = Qij - Pi,

where Pij means an image score of item i in a pupil i before learning the CAL material (I,) and Qij means an image score after Learning (f,). On comparing CAL courseware, 1/JZ i Al, was utilized. i=, when II pupils responded. Actually

i=l

was used, since seven items out of 20 were selected in order to calculate average AI, i.e. these 7 items were familiar, attractive, open-minded, good, easy, compatible and clear. Separately, six items were selected by the varimax rotation method upon the principal component solution of 20 items. These were ‘incompatible vs compatible”, and ‘difficult vs easy’ (from Familiarity factor); ‘bad vs good’ and ‘undesirable vs desirable’ (from Value factor); and ‘small vs large’ and ‘slow vs fast’ (from Affect factor).

Evaluation methods for improving CAL materials

285

(3) The state anxiety (S) for each CAL courseware was measured by the 15 item state anxiety scale during the learning situation. The six items in 7 grade multiple choice scale for evaluating state anxiety were selected through factor analysis of 15 items. These were ‘I feel bored’ and ‘I feel lack of motivation’ (from Saturation factor); ‘I feel receptive to learning’ and ‘I feel my confidence growing’ (from Interest factor); and ‘I feel rushed and ‘I feel confused’ (from Pressure factor). (4) The characteristics of CAL materials could also be found by the results of questionnaire (A)I which asked pupils the attitude towards CAL materials. The six items in the Yes-No scale were selected by the same method of factor analysis as the questionnaire I and S; two items from ‘Expectation’ factor, i.e. ‘I hope that the educational innovation would be achievable by the effective use of CAL’ and ‘I think both CAL and teacher instruction are necessary in school education’; two items from ‘Preference’ factor, i.e. ‘I think I would like to learn by such methods as CAL after graduation’ and ‘I prefer learning by CAL than by textbooks and guidebooks as it is more efficient’; and two from ‘Aversion’ factor, i.e. ‘I think that effective education is not achievable by computer’ and ‘I think CAL tends to inhibit creativity and originality’. by calculating the score of (5) The difficulties within the CAL materials could be distinguished the questionnaire for finding out the difficulties in each chapter. Where the score of an item j in a pupul i is Eij(_2 the average difficulty score of itemj

< Eij d 2)

in n pupils, that is: &jl =

i Eij/n i=l

could be gained. The larger the +jt, the more problematic

the material, and

could be utilized as the index of difficulty in each chapter, where 10 is number of items. For example & was 0.42 in chapter one and 0.16 in chapter three in Programme 4; and 0.24 in chapter one and 0.1 in chapter 2 in Programme 2. The six items composed of two items each from one of three factors could also be selected on the basis of the factor analysis, which was assumed to be also the minimum essential for evaluating the difficulties in each chapter. The selected items were ‘The instructions for tasks were understandable or not’ and ‘Adequate guidance concerning wrong answers was given or not’ (from Presentation factor); ‘Stimulation of learning occurred or not’ and ‘The learning process was enjoyable or not’ (from Stimulation of Learning factor); and ‘The content was generally difficult or easy’ and ‘The amount of work was tiring or not’ (from Content factor). (6) In the same way as 4jl and & in each chapter, eij =

Eij/n( - 2 d eij < 2)

i i=l

and

i=* OT

j-1

=

1On

(-2<4r$2)

could be calculated from pupils’ responses to questionnaire for finding out the difficulties in each course, where 10 is number of items. The six items were selected through factor analysis as follows: ‘The CAL is continuously available in.. hours’ and ‘I could learn for,. hours in CAL’ (from Continuity factor); ‘The way of thinking is constrained in CAL’ and ‘The CAL is troublesome’ (from Constraint factor); and ‘The sequence of learning was difficult to-follow’ and ‘I was bored because of the difficulty of understanding’ (from Construction factor). (7) The results of line connecting content analysis of CAL courseware could be utilized to analyze the merits and demerits of the CAL materials as shown in Fig. 2. In relation to Programme 3 they could be read that: for example, ‘As encouragement was interesting, I enjoyed it’; ‘As hints were rough, I was unsatisfied’; and so forth.

T. Item

to be evoluoted (1)

SAKAMOTQ

et al.

Attribute

Response (II)

KlIl

j fOb,ecttves of leornmq couid not

2 [ExPlonatlOn understand

3 fklaterials 4 fOuestion

Information

that could be gained

afterCALimple~ent~tio~

concerning learning history record

The various sorts of information that could be gained through the learning history records in CAL jmplementat~on are, for instance: total learning duration; number of correct and incorrect answers: total frame For learning; number of hints demands; number of unant~~~pated answers. These scores per number of correct answers could be derived. The useful information could also be obtained from the data analysis of history records. (1) By $ FLIST, in order to evaIuate the weight of each frame in the CAL materials. we could get a number of learners’ responses in each frame, the rate of responses among total number of learners in each frame and the average thinking duration in each frame. (2) By $ FHIST, we could obtain the distributions for each learners’ responses en each frame. These included the responses to function keys, i.e. Answer, Correct Answer, Call, Hint, etc. and the matching of responses to anticipated responses prepared in CAL courseware. (3) By $ CHAIN, we could get the table of each frame and the subsequent one and on which the actuaf transitive path graph would be gained as shown in Fig. 3; and the relative entropy of alternative paths to subsequent frames coufd be calculated for each frame. For example

where N means the number of paths from a frame, n means name of path and P,, means the probability of occurrence in path n. For example average relative entropy was 0.36 in Programme 2 arrd 0.75 in Programme 3. The relative entropy could in a sense be an index of compatibility with individual differences of pupils. When we compare these graphs and figures among different courseware modules, we can identify their characteristics. When we compare two sorts of transitive path graphs, one of which was gained directly from CAL courseware and the other was From the results of CAL impIementat~on after learning, we could easily find out the unused as well as frequently used frames and paths. On the basis of this range of information we cotrid improve the quality of CAL materials.

Evaluation Table 2. Final

version

methods

for improving

CAL materials

of the evaluation package proving CAL courseware

1. Before CAL Implementation 1.1 Checking CAL Materials 1.2 Analysis of Instructional Function Media Utilization in Each Frame 1.3 Transitive Path Graph 1.4 Average Branch Ratio 1.5 Linearity Ratio (1.6 CAL Simulation)

instrument

and

2. Immediately Before CAL Exercise (2.1 Questionnaire for Learning Motivation) 2.2 Questionnaire for Image Scale (I,) 2.3 Readiness Test 2.4 Pre-Test

I 3. During CAL Exercise 3.1 Questionnaire for Measuring State Anxiety (S) 3.2 Questionnaire for Finding Out the Difficulties in Each Chapter (EVA)

I 4. Immediately After CAL Exercise 4.1 Questionnaire for Finding Out the Difficulties in Each Course (EVB) 4.2 Post-Test 4.3 Questionnaire for Image Scale (12) 4.4 Questionnaire for Attitude Towards CAL (A) 4.5 Line Connecting Type of Content Analysis

5. After CAL Implementation Learning History Record 5.1 Total Learning Duration (D) . M & SD 5.2 Total Number of Passed Frames (PF) . , M & SD 5.3 Number of Correct (C) and Incorrect (Ic) Answers 5.4 DjPF 5.5 PFjC 5.6 Ic,‘C 5.7 Latent TimeiPF 5.8 Frequency of Use of the Function Keys

Analysis of Sequence 5.9 Actual Transitivre Path Graph 5.10 Actual Average Branch Ratio 5.11 Actual Linearity Ratio 5.12 Relative Entropy in Each Frame 5.13 Mean of Relative Entropy per Frame

Educational Effectiveness 5.14 Index of Educational 5.15 Ratio of Modification Evaluation Item

Effectiveness in Each

1 Question~ire

(see Table 3)

287 for im-

T. SAKAMOTO et al.

288 G1005 __

30 GOOIOA

Programme

=6

30~~"~7&"~"d-J~o*~_

GOOHA

G1006

Gl013

G1015

Gl018

G1022

G1401

G1402

G1403

5\-0-4

2

G1022A

KRCl3 66__. 26

K3005

03005

CLRl5

03005A

KRCl4

K3006

03006

CLRl4

100)

03006A KRCl5

KRC23

RC55

Programme

3

6 \, -9

KRC5 KRC 25

Fig. 3

Table 3. Questionnaire

(I)

incompatible difficult bad undesirable slow small

Value Affect

Saturation Interest Pressure

Expectation

Preference Attitude towards CAL(A) Yes or No Aversion

Presentation Difficulties in each chapter (EVA) 5 multiple choice

Stimulation of learning Content

Continuity Difficulties in each course (EVB) 5 multiple choice

Content analysis

Constraint Construction

instrument Item

Familiarity

Anxiety state (S) 7 multiple choice

package

Factor

Category

Image modification 5 multiple choice

in evaluation

I feel I feel I feel I feel 1 feel I feel

~ ~ ~ ~ ~ ~

compatible easy good desirable fast large

bored lack of motivation receptive to learning my confidence growing confused rushed

I hope that educational innovation would be achievable by the effective use of CAL. 1 think both CAL and the teacher instruction are necessarv in school eduction. I think I would like to learn by such methods as CAL after graduation. I prefer learning by CAL rather than by textbooks and guidebooks as it is more efficient 1 think that effective education is not achievable by computer. I think CAL tends to inhibit creativity and originality. The instructions for tasks were understandable or not. Adequate guidance concerning wrong answers was given or not. Stimulation of learning occurred or not. The learning process was enjoyable or not. The content was generally difficult or easy. The amount of work was tiring or not. The CAL is continously available in ___ hrs I could learn for __ hrs in CAL. The way of thinking is constrained m CAL. The CAL is troublesome. The sequence of learning was difficult to follovv. I was bored because of the difficulty of understanding. Lines connecting

types.

Evaluation

methods

for improving

289

CAL materials

we could obtain the learning results in matrix form, putting the standard sequence of frames in column and the learners in rows and showing the mean score, standard deviation, and disparity index in each frame.

(4) By $ EVAL.

SYSTEMATIC According as shown better

EVALUATION

to the above in Tables

mentioned

2 and

3. When

METHOD analysis,

FOR

we have

we use this

IMPROVING

developed

package

CAL

the evaluation

for improving

CAL

MATERIALS package

materials

instrument, we could

get

courseware.

We anticipate

that

for improving

the use of this package

CAL materials

can lead to the production

of higher quality courseware. Acknowledyrments-This project was undertaken with the cooperation of Professor H. Fujita, Mr K. Tamura, Mr H. Yoshikura, Mr M. Kobayashi, Mr T. Aoki, Mr M. Kawafune, Mr M. Abe and Mr N. Kasama. The authors express appreciation of their cooperation. The authors thank the Project Manager, Professor T. Horiuchi and Mr Itaya and Mr K. Yagi of JSPMI for their support. They also express their gratitude to Miss D. Laurillard, University of Surrey and Mr J. E. Proctor, NELP, for their helpful suggestions. REFERENCES 1. Hawkins A., The Perjix-mance and the Promise ofEaaluation in Computer Based Learning. International APLET Conference, Sheffield (1979). 2. Murphy R. T. and Appel I. L. R., Evaluation ofthe PLATO IV Computer Based Education System in the Community College, Princeton: New Jersey: ETS (1977).

APPENDIX A. ITEMS

OF THE

QUESTIONNAIRE

FOR

IMAGE

SCALE

1. Unfamiliar

Familiar

2. Unattractive

Attractive

3. Weak

Strong

4. Closed-minded

Open-minded

5. Superficial

Substantial

6. Inferior

Superior

7. Old

New

8. Noisy

Quiet

9. Rigid

Flexible

10. Obscure

Clear

11. Incompatible

Compatible

12. Slow

Fast

13. Bad

Good

14. Inequitable

Equitable

15. Undesirable

Desirable

16. Difficult

Easy

17. Ambiguous

Definitive

18. Small

Large

19. Cool

Warm

20. Wrong

Right

290

T.

B. ITEMS

OF

(7 multiple

choice

THE

QUESTIONNAIRE

scale---very

strongly

SAKAMOTO

FOR L

I

et al.

MEASURING

I

I

I

STATE

,

ANXIETY

, not at all)

neutral 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15.

I I I I I I I I I I I I I I I

feel feel feel feel feel feel feel feel feel feel feel feel feel feel feel

bored. tired. lack of motivation. conscious of other persons. the learning is enjoyable. my interest is growing. strained. I am concentrating. receptive to learning. my confidence is growing impatient. unsettled. self-possessed. confused. rushed.

C. ITEMS 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20.

OF THE

QUESTIONNAIRE

FOR

ATTITUDE

TOWARDS

CAL

(Yes or No)

I hope that educational innovation would be achievable by the effective use of CAL. I think that computers are much utilised in the medical field. I prefer to interact with computers rather than with human beings. I am able to consider logically the details of various events. I have little experience of computers. I think that mind is more important than matter. I think that the effective education is not achievable by computer. I think that highly civilized society makes human beings machine-like. The pollution in lakes and rivers comes about because of the fall in the level of the quality of the human mind. I think making cloths for oneself is unfashionable. I think that those subjects one dislikes would be better learned by using CAL. I think CAL is more appropriate for two persons co-operatively than one independently. I think the difference of levels of achievement will be increased by the extensive use of CAL. I think I would like to learn by such methods as CAL after graduation. I think that some subjects may be more suitable for CAL than others. I think both CAL and teacher instruction are necessary in school education. I think CAL tends to inhibit creativity and originality. I think one becomes gradually bored with CAL, even though one is interested at the early stage. I think CAL would be useful when preparing for entrance examinations. I prefer learning by CAL than by textbooks and guidebooks as it is more efficient.

D. ITEMS

OF THE QUESTIONNAIRE 1. The content

was generally: difficult

FOR

FINDING

OUT DIFFICULTIES

IN EACH

I

I

I

I

I

easy

I

I

I

t

I

slow

3. The meanings of the tasks were: comprehensible I

I

I

I

I

incomprehensible

I

1

I

I

not available

I

I

I

J

not understandable

I

J

not given

2. The pace of learning

4. Necessary

was: fast

hints were: available

I

5. The instructions

for the tasks were: understandable I

6. Adequate

guidance

7. The correct

8. The amount

9. Motivation

10. The learning

concerning given

wrong L

answers

was:

1

I

I

I

1

I

inadequate

I

I

I

I

I

not tiring

for learning: occurred

I

I

!

I

1

did not occur

process was: enjoyable

I

t

I

I

1

not enjoyable

answers given were: adequate 1 of work was: tiring

CHAPTER

Evaluation

E. ITEMS (5 multiple 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.

OF THE QUESTIONNAIRE choice scale)

methods

for improving

FOR FINDING

291

CAL materials

OUT THE DIFFICULTIES

The CAL was understandable in comparison with the conventional lecture. The sequence of learning was easy to follow. There is freedom to think about other information than the presented ones. The way of thinking is constrained in CAL. The learning was done at one’s own pace. I was bored because of the difficulty of understanding. The CAL is troublesome. I would like to do CAL in future. The CAL is continuously available in __ hours. I could learn for ~ hours in CAL.

IN EACH

COURSE