patient satisfaction: Development of a general scale

patient satisfaction: Development of a general scale

Evaluation and Program Plannmg, Vol 2, pp 197-207, 1979 0147-7189/79/030197-11 $02 00/0 Copyright © 1979 Pergamon Press Ltd Pnnted m the U S A All n...

907KB Sizes 0 Downloads 10 Views

Evaluation and Program Plannmg, Vol 2, pp 197-207, 1979

0147-7189/79/030197-11 $02 00/0 Copyright © 1979 Pergamon Press Ltd

Pnnted m the U S A All nghts reserved

ASSESSMENT OF CLIENT/PATIENT SATISFACTION: DEVELOPMENT OF A GENERAL SCALE DANIEL L LARSEN, C. CLIFFORD ATTKISSON, WILLIAM A HARGREAVES,AND TUAN D. NGUYEN Department of Psychiatry U n i v e r s i t y o f Calafornla, San F r a n c i s c o ABSTRACT The development and shaping o f a general scale to assess chent/pattent satisfaction is reported The scale, the CSQ, was constructed empirically by the authors The CSQ zs a response to several problems and issues that currently cloud the measurement o f consumer satisfaction m health and human servzce systems These problems and issues m assessmg satisfaction are described Finally, we present practical expmences to date m using the CSQ along with general psychometric quahttes o f the scale and correlations o f CSQ results with chent characteristics, service utilization, and service outcomes

In recent years there has been a significant shaft toward broadening the scope of client participation in the evaluation of human service programs A notable example of this trend is the proliferation of research on client and patient satisfaction. The dlstlngulshang feature of satisfaction research is that serwce recipients are explicitly asked to evaluate the serwces prowded to them The current paper discusses reasons for assessing satisfaction with services, outlines issues and methods in conducting thas type of evaluation, and presents our progress m developing a general scale for use in human serwce programs

Evaluations of human service programs incorporate a wide range of methods representing multiple interests and perspectives (Attkisson, Hargreaves, Horowltz, & Sorensen. 1978) However, until recently, the service consumer's viewpoint has often been ignored or under-represented Even when clients participate in the evaluative process, they are traditionally placed in the role of the ones being evaluated For example, whale many investigations of the outcome or effectiveness of services Include client ratings of their functioning or change In functioning, these ratings do not constitute direct evaluation, by the chents, of the program from which they receive servaces.

I S S U E S IN A S S E S S I N G C L I E N T S A T I S F A C T I O N Why Assess Chent Satisfaction ~ Many evaluators and researchers have advocated the inclusion of client sansfactlon ratings as one component of human service program evaluation (e g , Hargreaves & Attkisson. 1978 Margolls, Sorensen. & Galano, 1977, McPhee, Zusman. & Joss, 1972, Zusman & Slawson. 1972, Marvlt & Beck. Note 3) These writers have advanced several compelling reasons for assessing client satisfaction and, more generally. for involving the client in the evaluation of programs F~rst. when the client's perspective is not taken into account the evaluation of services is incomplete and biased toward the p~ovlder's or the evaluator's perspective In mental health it has often been observed that chents and therapists seldom agree on the amount of progress made m treatment (e g, Feffel & Eells, 1963, Horenstein. Hous-

ton, & Holmes, 1973, Strupp, Fox, & Lesser, 1969) This lack of agreement has often been used as a jusnficatlon for dismissing client ratings Current thanking, however, suggests that client and therapist ratings represent different perspectives, both of whach are needed to obtain a more complete assessment of service process and outcome (cf Attkisson et al., 1978, Strupp & Hadley, 1977). In other words, clients' ratings, though often at variance with those obtained from other information sources, still represent a potentially valid perspective (Waskow & Parloff, 1975) Second, in many human servace fields, there are legislative mandates to include clients and/or citizens in the evaluanve process In mental health, for example, the Community Mental Health Centers Amendments of 1975

Reques~ reprmt~ from C Chfford Attklsson Department of Ps)chlatr) Um~erslty of Callforma San Francisco, Cahforma 94143 197

198

L~RSEN

-kTTKISSON H ~ . R G R E ~ \ E S

(Title III of Pubhc Law 94-63) reqmre a broad-based evaluation of programs m order to receive continued funding One provision of PL 94-63 pertains to measunng the acceptability of services, which necessitates some form of client or patient participation in program evaluation The last, and perhaps the most compelling reason to assess client satisfaction is offered by Marvlt and Beck (Note 3, see also Margohs et a l , 1977) These writers point out that pubhcly funded health and human services in general are suppher dominated The poor, the nonmobfle, and other disadvantaged mtlzens have practically no alternatives to local public servme programs When m need, these citizens often cannot select alternative services based on consideratmns o f cost and quahty, for they are unable to go elsewhere for services, even if dissatisfied Furthermore, pubhcly funded organizations seldom have a profit motwe and have virtually no financial incentive to satisfy the client or to involve the client in the evaluative process If the client leaves the program, m o n e t a r y penalty ts rarely mcurred. Thus, without explicit client evaluation of services received, the d e t e r m m a t m n of service quahty, adequacy, and appropnateness is left in the hands of service prmaders and managers Marvat and Beck argue that this situation m a y lead to a phenomenon o f " u n d e r doctoring," that ~s, the provision o f too httle service, or service of poor quahty

and NGU~EN

1969) At the other extreme, some would accept the data at face value and proudl.~ proclaim, for example, that "80% of our clients report they are very' satisfied "" Both of these positions are shortsighted and m~sguided since the~e are approprmte ways to derive meaning and usefulness from client satisfaction data In the latter sections ot this paper, we will elaborate fully on these approaches

2 Lack of Meaningful Comparison Bases It is currently' difficult to find meaningful comparattve bases for interpreting client satisfaction findings. Levels of satisfaction in absolute terms and xn isolation from other data are meaningless For example, suppose that the mean satisfaction score for one service setting is 70 on a scale of 1 to 100 with a standard deviation of 10 What can one conclude 9 Without some basis for comparison, little can be stud Such statements as, "'In general, our clients are sansfled," may be misleading Now, suppose further that the mean satisfaction score on the same scale in a sample of comparable settings is 85 Clearly, a score of 70 is much lower than the mean for the population of similar programs With this additional reformation, we could now conclude that chents in the program under study are less satisfied, on the average, than clients m comparable programs

3 Lack of a Standard Sattsfacnon Scale Closely related

Problems in Collecting and Using Satisfaction Data Despite the above mentioned reasons for actwe involvement of health and human service consumers in program evaluation, there are sermus problems w~th using satlsfacn o n data m this process. These problems include (a) the high rate o f reported "satisfaction," (b) the lack of a standardized scale for assessing satlsfactmn, (c) the difficulty in obtaining unbiased samples, and (d) the low relevance of satlsfactmn data for program management

1 Hzgh Levels of Reported Sansfacnon A major problem encountered m using satlsfactlon measures relates to the ubiquitous finding that service recipients report l'ugh levels of satisfaction The mental health literature IS replete with such findings (e g , Denner & Halpenn, 1974, Frank, 1974, Gdhgan & Wdderman, 1977, Goyne & Ladoux, 1973, Henchy & McDonald, 1973). For example, Frank, Salzman, and Fergus (1977) noted a "preponderance of positive responses" among former psychotherapy outpatients Linn (1975), m his review of patient evaluation of health care, concluded that, from study to study, levels o f satisfaction are very l'ugh, regardless o f the m e t h o d used, the populatmn sampled, or the object of the rating. This finding o f a hagh level of satisfaction can be interpreted m several ways. At one extreme, it can be argued that such ratings should be dismissed as valueless, since they may be due solely to the chents' desire to gave "grateful testimonials" and to other demand characteriStlCS of the rating situation or instrument (cf. Campbell,

to the Issue of meaningful comparison bases is the lack of a standardized measure of client sansfactaon. A unique feature of client satisfaction research to date is the tendency of mvesngators to invent their own questionnaires or to modify existing scales m such a way that one is not sure whether the original and modffied versions measure the same thing (e g , Love, Caid, & Davis, 1979) Yet, it is obvious that one can not make vahd and meaningful compansons of different programs or components o f the same program when the data collection methods and instruments differ from one measurement m u a t m n to the next If one program uses a 10-Item scale whale another uses a 20-~tem scale and b o t h scaJ.es have httle m c o m m o n In terms of content and format, data from these two programs cannot be pooled to estabhsh normative data. Thus, It is essenttal to estabhsh a standardized scale as a sound basis for interprogram comparisons (Hargreaves & Attkasson, 1978, Strupp & Bergan, 1969, Strupp & Hadley, 1977) Even within programs, standardized measurement is necessary in order to compare different time periods, different groups of clients, or chents recmvmg different servme packages. Thus, at a minimum, when an existing scale is modified for use in a specific situation, one should establish clearly the psychometric relationships between the original and modified versmns of the scale

4 Dtfficulty m Avozdmg Samphng Bzases Another major difficulty in satisfaction research is obtaining a representative sample of the client population. When services

Assessment of Chent/Pataent Satxsfactmn Development of a General Scale are extended over time, dropout from the program may be a common occurrence This is certainly true m mental health outpanent programs (e g., Brandt, 1965, Garfield, 1971, Sue, McKmney, & Allen, 1976) Since &ssatlsfied chents are more hkely to drop out, the timing of data collecUon may deterrmne the extent and &rectlon of bins m the data. If data gathering takes place a long time after the chent entry point and those who drop out are not reached, then samphng bins ts greatly increased. If, on the other hand, data are collected close to the point of chent entry m order to counter the dropout bins, then chents have not experienced the complete serxnce package A common practice has been to sohclt ratings after chents have left the program. To some extent, this pracnce obvaates the dropout problem, since all chents are contacted However, because inexpensive data collection methods (e.g., marled questionnaires) must generally be used, one typlcaUy encounters the problem of low return rates, often below 35%. Again, the posslbdlty of samphng bins ~s very great Perhaps satisfied chents are more hkely to return the questionnaires Another possibility IS to sample chents cross-sectionally. For example, ratings could be sohclted from all chents who recewe services during a one-week period. This approach allows the samphng of chents who have had dafferent amounts of experience with the program. Tbas is often the least expenswe approach, but ~s biased against the mclusi3n of chents who drop out early or who miss theu appointments during the data collectmn period. These latter problems can be allewated by extending the data collectmn period. In general, the utlhty of different samphng points needs further study For comparisons over time and within the same program, any consistent samphng scheme is probably satisfactory But before client satisfaction data

199

are used for comparisons among programs, 1t lS essential to understand the relative biasing consequences of various samphng approaches If the dlrectmn and degree of bias of a samphng scheme can be shown to be consistent across all programs under consideration, then even a relatively large samphng bias may be tolerable If, on the other hand, the extent and direction of bias vary from one program or s~tuatlon to the next, then one must continue to search for samphng methods that minimize these biases 5 Htgh Cost and Low Relevance. From the viewpoint of program managers, methodological considerations may be far less important than other practical considerations First, there may be a lack of funds and human resources to conduct a soptustlcated or complex study Therefore, simple and inexpensive methods for assessing chent satxsfacnon are desxrable and, m many settings, an absolute necesmy. Second, there would appear to be little reason to assess client satlsfactmn ff the results wall not be used for program planning or demslon malang How can one fulfill samultaneously the requirements that chent sat~sfactmn assessment be s~mple, standardized, and usefuP A standard chent satasfactlon scale does not have to be long and comphcated. Using a small set of 1terns will decrease the cost of data collectmn, tabulatmn, and analysis The hkelLhood of the f'mdmgs being used can be increased by revolving potentml users in the planning of the study and by ehcltmg client evaluatmns of specific program aspects that need attention (see, e.g, Cahn, 1977, Note 1). Standard samfaction items can be supplemented by specially designed items addressing issues of pamcular local interest It ~s also useful to sohclt quahtatwe comments to supplement scaled items In this way it wall be possible to achieve useful assessments of chent sat~sfactlon that are simple and inexpensive.

ENHANCING THE USEFULNESS OF SATISFACTION DATA In the future, normative data enabhng comparisons between programs with slmdar chent populations may become possible At present, however, the greatest usefulness of satisfaction data resides m wIthan-program comparisons We will outhne below several strategies for such comparisons using chent satisfaction data 1. Focus on Dissatisfaction Data

One can attempt to identify (a) subgroups of &ssatlsfied chents within a program, (b) aspects of the service dehvcry system with wtuch clients are less sansfied, or (c) relative satisfaction of sImdar clients in alternate servxce modahtles One method, for example, involves an examinatmn of socmdemographlc correlates of satisfaction Quahtative comments may be solicited from the chents m order to ascertain the reasons for a particular subgroup's d~ssatisfaction with special focus on components or aspects of

the program that lead to client &ssatlsfactlon (e g., costs, accessibility, competence and attitudes of staff, outcome) Finally, one can assess the relative satisfaction levels of chents m &fferent service modalmes if clients are assigned randomly to alternative service modalmes, or when matctung strategies or covanance analyses are feasible for such comparisons In each of these instances, the strategy is to identify &ssatlsfied chent groups or apparently deftclent aspects of the service dehvery system In most sltuatmns, the results of a chent saUsfactaon survey wall be suggestive, and should be confirmed through management review or by more focused stu&es Denner and Halprm (1974) and Frank et al (1977) offered useful examples of sunple chent satisfaction surveys 2 Use Time-Series Analyses Client satisfaction can be monitored over t~me within a

200

L~RSEN

CTTKISSON H A R G R E ~ ' , E S and \ ( ; L ' : E N

program With this type of monitoring, one can examine the impact of sudden, major programmatic changes on clients" satisfaction by using quasi-experimental time-series analyses (cf Campbell, 1969, Hargreaves & Attklsson, 1978. Rlecken & Boruch, 1974) As a simple example. suppose that a mental health outpatient cllmc began a new counseling program focused on sexual dysfunction Clients with sexual problems could then be considered as belonging to the experimental group while chents with other presenting problems could serve as the comparison group Satisfaction data from the two groups could be collected over time both before and after the introduction of the new counseling program In this idealized example, suppose clients who do not present sexual problems show no change in satisfaction over time while chents with sexual problems show a consistently lower level of satisfaction when only tradmonal services were available but a consistently higher level of satisfactmn after the new program is implemented If one assumes that the two subpopulatlons are equivalent over time, except for sexual problems, then this finding would provide evidence for the success of the newly implemented program

3 Relate Satisfaction to Expectations Marvit and Beck (Note 3) have suggested that sat:sfaction be measured prospecnvely That Is, they argue that satisfaction should be defined as the extent to which client expectations regarding services are later fulfilled While the research hterature to date has not provided definitive support for the hypothesis that fulfilled expectations result in greater satisfaction (Duckro, Beal, & George, 1979), dissatisfaction resulting from unfulfilled

expectations is ~o~tll~ ol attention But it should be noted that clients e x p e c t a t : o n s m a y b e m m l m a l :napDoprlate, or unreahst:c partlcula:ly among some chent groups receiving mental health services (A~onson & Oxenall, 1966 Garfield & Wolpln, 1963 Ove:all & Monson 1963) When expectanons are appropriate ot reasonable, dissatisfaction suggests a need to rectlf? program deficiencies On the other hand, ~hen expectations are inappropriate, one needs to consider ways of altering these expectations (Albronda, Dean, & Starkweather, 1964 HoehnSaric et al, 1964 Mosby, 1972, Heflbrun, 1972, Orne & Wender, 1968. Schonfield et al 1969 Venema. 1972)

4 Triangulate the Measurement of Satisfaction Finally, the major bulk of satisfaction research to date has employed questionnaire measures Behavioral indices of satisfaction should also be explored. Possible measures include actual client recommendation of the program to other persons and chent dropout from the program, including clients who leave to go elsewhere for services because of dissatisfaction To date, research on behavioral measures of satisfaction has been sparse Simons, Morton, Wade, and McSharry (1978) have found that clients' actual recommendation of the program to others correlated only minimally with their indication on the questionnaire that they wouM recommend the program to someone else Concerning client dropout from services, Hargreaves, Showstack, Flohr, Brady, & Hams (1974) found that the program aspects that led clients to miss appointments were those that were also rated as least satisfying More work, then, is needed with these indices

THE CLIENT SATISFACTION As mentioned above, there 1s as yet no standardized measure of client satisfaction. Furthermore, there have been few attempts to determine whether and how much different aspects of the service delivery process affect dffferentmlly the overall clients' satlsfactmn with services. For example, chents may be very satisfied with the service received, but may be dissatisfied with its cost or the location of the facility Or, they may not differentmte aspects of the program and consequently may have only a very general feeling of sat:sfactmn or d:ssausfactlon McPhee et al. (19.75) argue for the former In general, the concept of patmnt satisfaction is too s:mphstlc Satlsfactmn with cost, accesslblhty, apparent professional skillfulness, staff-patient interactmn, and physical surroundings must all be taken into account in addmon to the tradltmnal concern with patients' satisfaction with outcome of service (pp. 401-402) Because a standard questionnaire is needed, we sought to develop a simple scale that could be used in a wide van-

QUESTIONNAIRE

ety of settings. In addmon, we investigated the degree to which clients can differentiate thmr expressions of satlsfactmn along multiple dimensions Initial Scale Development Our first step m developing the Client Satisfaction Questlonnmre (CSQ) was to consult pubhshed and unpublished sources in order to identify the potential determinants of satisfaction with serwces. From this hterature search, we were able to identify nine categories of possible determinants of satisfaction For each category we created nine 1terns. Our intent m the selection of these Items was to ensure breadth of content Each item was phrased as a question having a four-point anchored answer without the neutral posmon Table 1 lists the nine categories w~th an example o f an item in each. A group of 32 mental health professionals ranked the nine items in each category accorchng to how well they tapped the dimension in question Items were ranked from best (9) to worst (1) with no ties Items receiving a mean rank of 5 or higher were kept m the pool This

Assessment of Chent/Patlent Satisfaction Development of a General Scale

201

TABLE 1 CONTENT CATEGORIES AND SAMPLE ITEMS Category

Sample Item

Physmal surroundings

In general, how satmsfled are you with the comfort and attractiveness of our facdmty~

Support staff

When you first came to our program, did the recept=omsts and secretaries seem friendly and make you feel comfortable~

Kind/type of service

Considering your particular needs, how appropriate was the kind of servme you recelved~

Treatment staff

How competent and knowledgeable was the person with whom you worked most closely7

Quahty of serwce

How would you rate the quahty of serwce you recelved~

Amount, length, or quant=ty of serwce

How satMsfled are you with the amount of help you recewed~

Outcome of serwce

Have the serwces you recmved helped you to deal more effectwely with your problems~

General satisfaction

In an overall, general sense, how satisfied are you with the serwce you rece=ved~

Procedu res

When you first came to our program, were you seen as promptly as you felt necessary?

left 45 items with a m l m m u m o f 4 and a m a x u n u m o f 6 per category. The reduced pool was then rated by 31 members from various Cahfornla County Mental Health Advisory Boards. These raters were asked, gwen their p o s m o n as advasory personnel, to rank items (again, within category) by selecting those about which they would most hke to receive feedback The three top-ranked items in each category were selected Four additional items were also retained because their content was sufficiently different to justify inclusion Thus, the preliminary version o f the scale was composed of 31 Items, wIth a mmmaum of three Items m each category (A copy of this preliminary version can be obtained from the second author ) This preliminary scale was administered to 248 mental health clients in five service settings The sample, described in Table 2, was exclusively outpatient, but included persons receiving a variety o f treatments As seen In Table 2, some individuals were still in treatment at the tune they completed the prehmmary scale, while others had left treatment for periods o f up to six months As in previous research, the distribution o f responses (based on the total score) was skewed, with a large propo m o n of "satisfied'" clients and smaller numbers of "dissatisfied" chents, whose scores had a much wider range The data from this prehmlnary stud)" were submitted to a principal-components factor analy sis, using squared multiple correlations as initial communahty estimates (Nle, Hull Jenl~ns Stembrenner & Bent 1975) The first fac-

tor derived from this solution accounted for 43% o f the total variance and roughly 75% o f the c o m m o n variance. Even when Items with high first-factor loadlngs were removed and the analysis repeated, no other factor accounted for as much as 10% o f the total variance. In the mare analysis, the second factor accounted for less than 7% of the c o m m o n variance This finding suggests that only one salient dimension emerged from responses to the items m the prehmlnary scale. While this finding may not hold for client satisfaction ratings m general (e.g., Love et al., 1979), it does seem to be true of responses to the present scale The Final Scale To construct a briefer scale for assessing general satisfaction with services, the factor loadlngs and item - total correlations were examined Eight items were selected which loaded highly on the unrotated first factor and which exhibited good rater-Item and item - total correlations The inter-Item correlations for this finalized CSQ are presented m Table 3 for two independent samples of outpatient clients Coefficient alpha (cf Nunnally, 1967) for this final CSQ is 93, Indicating that it possesses a high degree of Internal consistency. In other words, the eight items provide a homogeneous estimate o f general satisfaction with services A copy of the CSQ is reproduced m Table 4 Items 3, 7 and 8 m Table 4 also appear to function well as a smaller global measure o f satisfaction.

202

LARSEN

4,TTKISSO\ I - f 4 R G R E ~ \ E S and NGL'~EN

Correlates of Sansfaction

TABLE 2 CHARACTERISTICS OF THE SAMPLE Vamable

Percentage at level

Education <" Htgh School Htgh School > H~gh School

41 32 27

Ethmc group Wh ite Nonwhite a

63 37

Employment status Employed Unemployed Not in Job Market

42 38 20

Outpatient ws~ts 1-5 6-10 > 10

51 22 27

Soc=al class b Upper (I-II) Middle ( I l l - I V ) Lower (V)

22 43 35

Prewous treatment m same program~ Yes No

41 59

Sex Male Female

39 61

Yearly income $ 2,500 & Under $ 2,501-$ 5,000 $ 5,001-$ 7,500 $ 7,501-$10,000 $10,001 -$12,500 $12,501 & Above

22 19 18 15 15 11

Servme fee Full Fee Partml Fee No Fee

31 31 38

Marital status Never married Married Separated, w~dowed, d~vorced

33 36 31

Age at adm~ssmn Under 21 21-40 Over 40

12 64 24

Stdl m treatment7 Yes No

42 58

Prevtous treatment elsewhere7 Yes No

33 67

aAImost excluswelv black bHolhngshead mdDces

The relanonshtps between the CSQ and the ~armbles m Table 2 were tested using ch>squa:e statistics and correlations In the ch>square tests CSO scores were collapsed into three levels low (8-20). m e d m m ~21-2o). and h:~: (27-32) The results indicated that sattsfacnon ~as not slgmficantly related to years of education famdy mcome. marital status, amount of service, age at admission. social class, or previous treatment at another facility All other variables showed slgmflcant relatlonsh:ps with sansfaction, as explained below Nonwhite chents were proportionately less satisfied than were white chents (X: (2) = 4 87. p < 05) Women were more polarized than men m thmr responses (X= (2) = 11 18, p < 005) That ts men tended to respond m the middle ranges, whereas women gave p r o p o m o n a t e l y more very posture and negative responses Chents who were unemployed were less sat:stied with serwces than either those who were employed or who were not in the job market 0( 2 (4) = 11 83, p < 05) Persons still in treatment were more sansfied than those who had left treatment (X 2 (2) = 12 82, p < 01) Chents w:th a prevmus treatment eptsode in the program were less satisfied (X 2 (2) = 6 67, p < 05) This latter finding is somewhat surpnsmg, but may be due to the fact that persons returning frequently for treatment have more serious or chromc problems Finally, chents paying a partial fee were more sat:stied than those who paid full fee or no fee (X 2 ( 4 ) = 18 5 3 , p < 001) In general, the correlates of chent satlsfactmn In the sample make sense lntumvely Only the finding regarding response style differences between males and females can be regarded as somewhat peculiar It may be an artifact o f the sample or a genuine difference m response style. Further work is needed on this point

Further Testing of the Client Satisfaction Questionnaire The 8qtem CSQ was included m an e x p e n m e n t a l study of t h e r a p y , o u t c o m e m an urban C o m m u m t y Mental Health Center (Larsen, 1977). The questionnaire was admlmstered to 49 outpatient chents approximately four weeks following admlss:on for mdlv:dual therapy Three chents could not be reached at followup. The majority of chents filled out the CSQ following a regular chmc v:slt and the remainder were sent the questlonnatre. The 8-1tern scale, as it performed m this study o f psychotherapy outcome, retained the same basic psychometnc propertms Item means were slmdar and the coeffic:ent alpha was 92. Relatlonslups o f CSQ scores to other dependent measures m the study (Larsen, 1977) are presented m Table 5. Two vanables among the general ratings and chent characteristics correlated significantly with satlsfactzon. Chents dropping out o f the program within the first m o n t h tended to be less satisfied than those still in the program (r = .37,

Assessment of Client/Patient Satlsfactmn Development of a General Scale

203

TABLE 3 INTER-ITEM CORRELATIONS INTWO SAMPLES OF MENTAL HEALTH OUTPATIENTS Item N u m b e r a Item

1

2

3

4

5

6

7

8

55 .66 55

64 63

.68

-

51 71 39

-76 59

64

Sample #1

1 2 3 4 5

6 7 8

71 66 64 61 68 78 72

.65 57 .62 67 71 62

-

49 .56 .65 66 85

-

54 .56 69 79 Sample Cf2

1 2 3 4 5

6 7 8 aRefers

67 61 .69 67 70 79 51

67 67 48 .59 68 54

52 .63 .65 71 41

.56 46 67 61

to the =tem number m Table 4

p < .01) Among chents still m treatment, those rmssmg a greater percentage of thelr scheduled appointments also tended to be less satisfied with services (r = .27, p < .06) Unhke the findings m the original sample, no relationships between satisfaction and sex or ethmc group were observed Ttus was true even when CSQ scores were blocked and chl-square tests performed on the data Chents made a global self-rating of improvement at the four-week followup and completed three subscales of the Symptom Checkhst (SCL-90, [Derogatls, Llpman, & Cow, 1973]) at admlssmn and followup Self-raUngs of global Improvement correlated slgmficantly w~th satisfaction scores (r = 53,p < 001) At followup, two of the SCL-90 subscales, depressmn and anger, showed a low but slgmficant (p < 05) and negative correlatmn with satlsfactmn, when the relatlonstup to admission scores was pamalled out (r = 32 and 36) Thus, chent-rated therapy gain is correlated with chent satlsfactmn, but apparently less so as the measure becomes more specific (vs global). Therapists also made several ratings at the two measurement points Thmr ratings of chent global improvement were not significantly related to sausfactmn Followup scores on the Global Assessment Scale (Endicott,

Spltzer, Flelss, & Cohen, 1976) were also unrelated to CSQ scores Partial correlations were computed between foUowup scores on a modified and abbreviated version of the Bnef Psycbaatrlc Rating Scale (BPRS, [Overall & Gorham, 1962]) and satisfaction, controlling for admission levels Three of the nine symptom ratings (anxiety, thought disturbance, and lnterpersonal-soclal-mantal problems) correlated slgmficantly (p < 05) with client satisfaction (r = .37, .33, and .33, respectively) Two other ratings, depression and job-related difficulties, showed a marginal (p < 10) relationsbap (r = 28 and .29, respectively). The total score on the modified BPRS correlated 44 (p < 01) with the CSQ. Thus, agmn, there IS the suggestion of a modest relationship between therapy gain and satisfaction. Therapists were also asked to rate their satisfaction with thmr work with the chent This rating correlated 42 (p < 01) with the client satIsfactmn ratings Finally, therapists' estimates of how satisfied they beheved the client to be were correlated 56 (p < 01) with the actual client rating on the CSQ The latter finding prowdes some ewdence of the scale's concurrent vahdlty

204

L~.RSE\

~TTKISSO\

H 4 . R G R E ~ . \ E S and \ G L ~ F \

TCBLE 4 THE CLIENT SATISFACTION QUESTIONNAIRE (CSQ) Please help us ~mprove our program by answering some questions about the serwces y o u have received at the We are interested m y o u r honest oplntons, w h e t h e r t h e y are positive or negattve Please answer a / / o f the questions We also welcome y o u r comments and suggesttons T h a n k y o u very much, we apprec[ate y o u r help CIRCLE YOUR ANSWER 1 H o w w o u l d y o u rate the q u a l t t y of serwce y o u recmved# 4 3 2 1 Excellent 2

3

4

No, n o t really

Yes, generady

Yes, defm=tely

2

1

Most of m y needs have been met

O n l y a few of m y needs have been met

None of m y needs have been met

If a friend were m need of slmdar help, w o u l d y o u recommend our program to h~m/her? 1 2 3 No, d e f i m t e l y n o t

5

Poor

To w h a t e x t e n t has our program met y o u r needs? 4 3 A l m o s t all of m y needs have been met

4

Fmr

O~d y o u get the kind of serwce y o u wanted? 1 2 No, d e f i n i t e l y not

*3

Good

No, I d o n ' t t h i n k so

Yes, I t h i n k so

Yes, defmttety

3

4

M o s t l y satisfied

V e r y satisfied

H o w satisfied are y o u w i t h the a m o u n t of help y o u recetved? 1 2 Qwte dissatisfied

I n d i f f e r e n t or m i l d l y dissatisfied

Have the serwces y o u recmved helped y o u to deal more e f f e c t w e l y w~th y o u r problems? 4 3 2 Yes, t h e y helped a great deal *7

No, t h e y really d i d n ' t help

No, t h e y seemed to make things worse

In an overall, general sense, h o w satisfied are y o u w i t h the serwce y o u received? 4 3 2 Very sattsfled

*8

Yes, t h e y helped somewhat

Mostly satisfied

Ind=fferent or m t l d l y dlssat~sfted

If y o u were to seek help again, w o u l d y o u come back to our program? 1 2 No, d e f l m t e l y n o t

No, I d o n ' t t h i n k so

Qutte dtssattsfled

3

Yes, I t h i n k so

Yes, d e f l m t e l y

WRITE COMMENTS BELOW

*Can be used as a shorter scale

GENERAL DISCUSSION In summary, the CSQ appears to be a useful measure o f general satisfaction with services It possesses a high degree of internal consistency and correlates with ther-

aplsts' estimates o f client satisfaction Cox, Brown, Peterson, & Rowe (1978, Note 2) recently used ttus CSQ m a state-wide assessment of mental health service outcome

Assessment of Chent/Patlent Satlsfactmn Development of a General Scale

205

TABLE 5 C O R R E L A T E S O F CLIENT S A T I S F A C T I O N Variable

ra

pb

37 -17

01

General Measures Therapy dropout m first month Sex (1 = female, 2 = male) Age at admission Years of education Famdy income Social class index Percent of appointments kept Ethnic group (1 = white, 2 = nonwhite) Previous treatment (1 = yes, 2 = no)

- 05 10 - 09 -15 .27

06

-10 14

Client Ratingsc Symptom Checklist A n x i e t y subscale Symptom Checklist Depression subscale Symptom Checkhst Anger subscale Global =mprovement rating

15 32 36 53

05 01 001

Therapist Ratingsd Global Assessment Scale (GAS) Global improvement rating Brief Psychiatric Rating Scale (BPRS) Total score Somatic concern or complaints A n x m t y or tension Depression Anger or hostdlty Sleep or appetite disturbance Inappropriate affect Disturbance of thought Interpersonal-social-marital problems Job-related difficulties Therapist satrsfaction Estimate of chent satisfaction

16 .21

m

44 19

01

.37

.05

.28

10

23 .21 23 33 .33 29

42 56

w

.05 05 10 005 001

apearson correlations bAll probabdrttes are two-tailed CFrrst three are partial correlations dFor the GAS and BPRS, partial correlations were used

In that s t u d ) , the scale's coefficient alpha was 90 when used with chents after their first service contact (n = 213), and 94 when used in a 90-dab followup with the same clients (n = 113) On the other hand, the CSQ has been found to bear only moderate to low relationstups with measures of outcome after a very brief or short-term followup interval (Larsen, 1977 see also Cox et al , Note 2) This latter finding suggests that clients ma5 be able to differentiate between satisfaction with treatment and gmn from treatment

The CSQ also correlated with early treatment dropout, wtuch suggests that chent satisfaction may be used as a variable m a set o f d r o p o u t predictors Further work is needed to test, this possibility and to determme any relationship between chent sansfactlon, measured retrospectively, and the extent o f fulfillment o f pretreatment expectations as suggested by Marwt and Beck (Note 3) Finally, the CSQ can easily be supplemented by openended quesnons and/or items of special Interest to a parncular serxnce program, without undue rime demand on

206

L~RSIZN

~TTKISSO\

H~RGRE~\ES

c h e n t s Our e x p e r i e n c e s h o w e d t h a t it takes a c h e n t f r o m 3 to 8 m i n u t e s to c o m p l e t e the CSQ and one a d d m o n a l o p e n - e n d e d q u e s n o n Thus, this CSQ can be used to facahtate r a t e r - p r o g r a m and s a m p l e - t o - s a m p l e c o m p a r i s o n s Add m o n a l q u e s t m n s can be a p p e n d e d to test n e w 1terns, to

and \ G U ' t E X

increase the r w h n e s s o ( t h e m f o r m a t ~ o n o b t a i n e d to encourage m o r e specific f e e d b a c k f r o m c h e n t s and to enh a n c e the value and use o f results Despite its s u n p h c l t ) , the a s s e s s m e n t o f client s a t i s f a c t i o n , using a b r i e f s t a n d a r d scale, p r o m i s e s to be a useful evaluative t o o l

REFERENCE NOTES Research, DP,lSlOn of Analysis and Information Services and the Dlvlsmn ot Mental Health, Department of Health and Socml Services, 1968

CAHN J Gluing responslblhty for evaluatmn to consumers A techmque for improving competence m human servxce delivery systems Paper presented at the Annual Meeting of the Evaluatmn Research Socmty, Washington D C , October, 1977 3 COX, J B , BROWN T R , PETERSON, P D , & ROW, M M The two-state collaborative mental health outcome study State of Washington (Vol i) Olympm, Washington Office of

MARVIT. R C , & BECK G D Toward consumertsm m pubhc mental health servtces 4re consumers bemg trtcked or treated, and how shall we know ~ Unpubhshed manuscript Department of Health, State of Hawan

REFERENCES ALBRONDA, H , DEAN, R , & STARKWEATHER, J Socxal class and psychotherapy Archzves of General Psychtatry, 1964, 10, 276-283 ARONSON, H , & OVERALL, B Treatment expectatmns of patients m two socml classes Social Work, 1966, 11, 35-41 ATTKISSON, C C , HARGREAVES, W A , HOROWlTZ, M J , & SORENSEN, J E (Eds) Evaluatton of human servtceprograms New York Academic Press 1978 BRANDT, L W Studies o f " d r o p o u t " patmnts m psychotherapy A revmw of findings Psychotherapy Theory, Research and Practtce, 1965, 2, 6-12 CAMPBELL, D T Reforms as experiments American PsychologTst. 1969, 24, 409--429 DENNER, B , & HALPRIN, F Measuring consumer satlsfactmn m a commumty outpost Amertcan Journal of Commumty Psychology, 1974, 2, 13-22 DEROGATIS, L , LIPMAN, R , & COVI, L The SCL-90 An outpatmnt rating scale (Prehmlnary report) Psychopharmacology Bulletm, 1973, 9, 13-28 DUCKRO, P , BEAL, D , & GEORGE, C Research on the effects of dlsconfirmed chent role expectatmns in psychotherapy A cntlcal revtew Psychologwal Bulletm, 1979, 86, 260-275 ENDICOTT, J., SPITZER, R L , FLEISS, J L , & COHEN, J The Global Assessment Scale A procedure for measuring overall seventy of psy'chmtnc disturbance Archives of General Psychzatry, 1976, 33, 766-771 FEIFEL, H , & EELLS, J Patmnts and therapists assess the same psychotherapy Journal of Consultmg Psychology, 1963, 27, 310318

GARFIELD, S L Research on chent variables m psychotherapy In A E Bergm & S L Garfield (Eds ), Handbook of psychotherapy and behawor change An emptrtcal analysts New York Wiley, 1971 GARFIELD, S L , & WOLPIN, M Expectatmns regarding psychotherapy Journal of Nervous and Mental Dtsease, 1963, 137, 353-362 GILLIGAN, J F , & WILDERMAN, M A An economical rural mental health consumer satisfaction evaluatmn Commumty Mental Health Journal 1977, 13, 31-36 GOYNE, J , & LADOUX, P Patients' opinions of outpatmnt chmc services Hospttal and Commumty Psychtatry, 1973, 24, 627-628 HARGREAVES, W A , & ATTKISSON, C C Evaluating program outcomes In C C Attklsson, W A Hargreaves, M J Horowltz, & J E Sorensen rEds ), Evaluanon of human servtce programs New York Academm Press, 1978 HARGREAVES, W A , SHOWSTACK, J , FLOHR, R , BRADY, C , & HARRIS, S Treatment acceptance following retake assignment to mdivldual therapy, group therapy, or contact group Archives of General Psychtatry, 1974, 31, 343-349 HEILBRUN, A Effects of briefing upon chent satisfaction with the lnmal counsehng contact Journal of Consulting and Clmtcal Psychology, 1972, 38, 50-56 HENCHY, T , & MCDONALD, L A global assessment of the effectiveness of a commumty mental health center through the use of a consumer questaonnalre, exChange, 1973, 1, 19-21

Evaluanon, 1974, 2, 20-21

HOEHN-SARIC, R , F R A N K , J D , I M B E R , S D , N A S H , E H , STONE, A R , & BATTLE, C C Systematic preparatlon of panents for psychotherapy I Effects on therapy behavmr and outcome Journal of Psychtatrw Research, 1964, 2, 267-281

FRANK, R , SALZMAN, K , & FERGUS, E Correlates of consumer sattsfactlon with outpatmnt therapy assessed by postcards Commumty Mental Health Journal, 1977, 13, 37-45

HORENSTEIN, D , HOUSTON, B K , & HOLMES, D S Chent's, therapist's, and judges' evahiaUons of psychotherapy Journal of ConsultmgPsychology, 1973, 20, 149-153

FRANK, R A A postcard survey assesses consumer satxsfactmn

Assessment of Chent/Pataent Satisfaction

D e v e l o p m e n t o f a G e n e r a l Scale

207

LARSEN, D L Enhancmg chent utihzanon o f commumty mental health outpatient services (Unpublished doctoral dissertation ) Umverslty of Kansas, 1977

REICHEN, H W , & BORUCH, R F (Eds) Soczal experimentation A method for planning and evaluating social intervention New York Academic Press, 1974

LINN, L S Factors assocmted with patient evaluatmn of health care Mzlbank Memorial Fund Quarterly (Health & Society), 1975, 53, 531-548

SCHONFIELD J, STONE, A , HOEHN-SARIC, R , IMBER, S . & PANDE, S Patient-therapist convergence and measuresoflmprovemerit in short-term psychotherapy Psychotherapy Theory, Research and Practice, 1969, 6, 267-272

LOVE, R E , CAID, C D , & DAVIS, J r , A The user satisfaction survey Consumer evaluation of an inner city commumty mental health center Evaluation & the Health Professions, 1979,2, 42-54 MARGOLIS, R B , SORENSEN, J L , & GALANO, J Consumer satlsfactmn in mental health delivery servaces Professzonal Psychology, 1977, 8, 11-16 MCPHEE, C B , ZUSMAN, J . & JOSS, R H Measurement of patient satisfaction A survey of practices in community mental health centers Comprehensive Psychiatry. 1975. 16, 399404 MOSBY, R Alteratmn of clients' expectatmns about cousehng m the direction of client-counselor mutuality by means of an experimental Intervention procedure (Unpublished doctoral dissertation University of Texas at Austin, 1971 )Dtssertatton Abstracts lnternatzonal, 1972, 33, 446B-447B (University Microfilms No 72-19635 )

SIMONS, L S , M O R T O N , T L , W A D E , T C , & MCSHARRY, D M Treatment outcome and followup based on chent case records in a mental health center Journal of Consulting and Chmcal Psychology, 1978,46, 246-251 STRUPP. H H , & BERGIN, A E Some empmcal and conceptual bases for coordinated research in psychotherapy A critical review of issues, trends, and evidence International Journal of Psychiatry, 1969, 7, 18-90 STRUPP, H , FOX, R E , & LESSER, K Panents view their psychotherapy Baltimore Johns Hopkins, 1969 STRUPP, H H , & HADLEY, S W A tripartite model of mental health and therapeutic outcomes With special reference to negative effects in psychotherapy American Psychologist, 1977, 32, 187-196

NIE, N H , HULL, C H , JENKINS, J G , STEINBRENNER, K , & BENT, D H SPSS Statistical package for the social sciences (2nd ed ) New York McGraw-Hill, 1975

SUE, 'S, MCKINNEY, H L., & ALLEN, D B Predictors of the duration of therapy for clients In the community mental health system Community Mental Health Journal, 1976, 12, 365-375

NUNNALY, J C Psychometric theory New York McGraw-Hill, 1967

VENEMA, J The effects of expectancy training, commitment. and therapeutic conditions upon attrmon from outpatient psychotherapy (Unpubhshed doctoral dissertation Fuller Theological Semmary, 1970 ) Dissertation Abstracts Internatzonal, 1972, 32, 6664B-6665B (University Mlcrof'flms No 72-15871 )

ORNE, M J., & WENDER, P H Anticipatory soclahzatlon for psychotherapy Method and rationale Amertcal Journal of Psychiatry, 1968, 124, 1202-1212 OVERALL, B , & ARONSON, H Expectations of psychotherapy in patients of lower socmeconomlc class American Journal of Orthopsychtatry, 1963, 33, 421--430 OVERALL, J E , & GORHAM, D R The Brief Psychmtnc Rating Scale PsychologzcalReports, 1962.10, 799-812

WASKOW, I E , & PARLOFF, M B (Eds) Psychotherapy change measures (DHEW Publication No ADM 74-120 ) Washington, D C U S Government Printing Office, 1975) ZUSMAN, J , & SLAWSON, M R Service quality profile Development of a technique for measuring quahty of mental health setraces Archives o f General Psychiatry, 1972, 22, 692-698