|
Questionnaire and Rating Scale Construction Tips
Compiled by
Raoul A. Arreola, Ph.D.
Developing Questionnaire Scales
Step 1. Identify the program objectives for which the questionnaire is being prepared; determine what specific information you hope to obtain from the questionnaire.
Step 2. Choose a response format. There are four basic formats: checklists, two-way questions, multiple-choice questions and ranking lists.
Examples:
Checklist: Which of the following journals in the library have you read in the last six months? (check as many boxes as necessary)
[ ] Journal of Irreproducible Results
[ ] Worm Runner's Digest
[ ] Women's Wear Daily
Two-Way Questions: Please read the descriptions of Programs A and B. If you were to be assigned to only one program, which would you prefer?
[ ] Program A [ ] Program B
Multiple-choice: There are several possible response scales for use with the multiple choice format. The following are examples of response scales for measuring agreement, frequency, importance, quality, relative quantity and likelihood. The most popular scales are shown in boldface type.
Agreement Scales
Frequency Scales
Importance Scales
Quality Scales
Relative Quantity Scales
Likelihood Scales
Ranking Scales: How do you like the following subjects in your program? Place a "1" next to the one you like the best, a "2" next to the one you like the second best, and so on. The one you like the least of the five subjects should have a "5" next to it.
[ ] Biology
[ ] Biochemistry
[ ] Physiology
[ ] Psychology
[ ] Research Design
Step 3. Identify the frame of reference of the respondents. Ask yourself - what vocabulary would be appropriate to use with this group? How well informed are the respondents likely to be? Does the group have a particular perspective that must be taken into account - a particular bias?
Step 4. Write the question
Step 5. Prepare a data summary sheet. (Consider using an optically scannable answer sheet).
Step 6. Critique the questions; try them out and revise them. Be careful not to use emotional language or negative wordings. Examples:
- Should supervisors forbid staff to leave the work area without permission?
- (Better: Should supervisors require that staff ask permission before leaving the work area?)
-
- Would you prefer that your child not take part in the program?
- (Better: Would you like your child to take part in the program?)
Step 7 Assemble the questionnaire. The appearance of a questionnaire is extremely important. The first impression it leaves will affect response rate.
Step 8. Administer the questionnaire. Plan on follow-up procedures to enhance the probability of a good response rate. Consider, for example, phone calls to those who have not responded - or, at least a second mailing of the questionnaire.
Developing Attitude Rating Scales
An attitude rating scale yields a single score that indicates both the direction and intensity of a person's attitude, although some scales yield "sub-scale" scores as well. For example, a rating scale designed to measure attitude toward a faculty member's teaching may have a set of items that relates to attitude toward the instructor's enthusiasm, another set that relates to the instructor's organization, another set that relates to the instructor's presentational style, and so on.
The Likert (Agreement) Scale
The most common agreement scale consists of a series of attitude statements. These statements embody extreme statements, either clearly favorable or clearly unfavorable. The agreement scale achieves a wide range of scores by having respondents report the intensity of an attitude. This is accomplished by providing gradations within the response alternatives. The respondents are asked to indicate their agreement with each statement on a 5-point scale:
SA = Strongly Agree A = Agree U = Undecided D = Disagree SD = Strongly Disagree
This question format is a familiar one that can be found in a variety of measures, not just attitude rating scales. It is a popular multiple-choice format that is frequently used in the construction of many types of attitude questionnaires. However, care should be taken to ensure that the statements being responded to have a real, plausible middle or "undecided" condition. If not, a 4-point scale omitting the "U = Undecided" response category should be used. (See other common agreement response scales above).
Steps for constructing and using an Agreement Scale.
1. Accumulate a large number of clearly favorable or clearly unfavorable statements about the attitude you wish to measure (approximately 60). As with the statements for the ordered scale, a good source might be prospective respondents.
2. As a pilot group (50 or more) to respond to these statements. The pilot group should consist of people who are (a) similar to the people whose attitudes you wish to measures, and (b) likely to express the whole range of attitudes you wish the instrument to detect.
3. Score responses by assigning them from on to five points - five for most favorable, one for least favorable. This, of course, means responses will be scored differently depending on whether the statement reflects a negative or positive attitude. Strongly agreeing with a positive statement would result in a score of "5" as would strongly disagreeing with a negative statement.
4. Compute a score for each respondent by totaling the points corresponding to his or her responses.
5. Identify high scorers (top 25%) and low scorers (lowest 25%).
6. Analyze each statement according to how high and low scorers responded to it to determine which items are discriminating well. Use an item analysis to accomplish this.
7. Retain those items (approximately 20) which provided good discrimination between high and low scorers.
8. Construct the questionnaire by listing the retained statements in random order.
9. Administer the instrument.
10. Compute a score for each respondent by totaling the scores corresponding to his or her responses.
Cautions in Constructing a Rating Scale
In the examples above we have used a Strongly Agree - Strongly Disagree type scale. Other scales which are sometimes used are shown in the table above. However, there may be occasions when you may wish to construct your own response scale in order to reflect a specific type of question you are answering. There are four main cautions that you should observe in constructing your own scale.
1. Each point on the scale should be defined.
Notice in the table above every point on each scale has a one or two-word definition. It is important in constructing your own scale that every point have a clear definition. The definition may be one or two words or it may be a longer statement. In any event, make certain that the respondent knows what each point on the scale represents. DO NOT leave it to the respondent to "make up" their own scale points. For example the scale below only has two defined points:
LO * * * * * HIGH
or
LO _ _ _ _ _ _ _ _ HIGH
Each respondent would have to make up their own definition as to what one asterisk or dash above LO means, or two asterisks or dashes below HIGH means. Their definitions may not be the ones you intend.
2. Use letter abbreviations to represent the points on a scale. DO NOT use numbers.
Sometimes you may see a scale such as the Strongly Agree - Strongly Disagree scale shown above used in the introduction to a questionnaire but the actual response positions use numbers instead of letter abbreviations. For example:
"Use the following scale to respond to the items below."
1 = Strongly Agree
2 = Agree
3 = Disagree
4 = Strongly Disagree
Item #1: The teaching in this course was effective. 1 - 2 - 3 - 4 - (circle one)
Items #2: The examinations were too difficult.- - - 1 - 2 - 3 - 4 - (circle one)
In the example above, when the respondent indicates their answer to an item they circle or mark a number representation. The reason we try to avoid this situation, especially in educational applications, is that numbers (especially the number range 1 - 4) take on a life of their own. The numbers tend to be interpreted as indicators of grades or desirability rather than an expression of, in this case, agreement or disagreement.
3. Make certain the scale is balanced.
In constructing your scale make sure you have as many positive response positions as negative ones. For example:
- (+) Strongly Agree
- (+) Agree
- (- ) Disagree
- (- ) Strongly Disagree
-
- - OR -
-
- (+) Strongly Agree
- (+) Agree
- (0) Undecided
- (- ) Disagree
- (- ) Strongly Disagree
If your scale has an even number of responses make sure half are positive and half are negative. If you decide to have a middle position make certain that it is a TRUE middle position that is neither a negative nor a positive response.
An example of a common UNBALANCED scale sometimes used in the evaluation of performance is:
Excellent (+)
Outstanding (+)
Satisfactory (+)
Unsatisfactory (-)
In scales such as these there is a tendency among respondents to somehow balance the scale in their minds. Thus, the "Satisfactory" response position tends to become associated with a negative performance - even though this may not have been the intention of those constructing the scale.
Another problem with the scale above is that its response positions are not parallel. The "Satisfactory - Unsatisfactory" responses belong to an essentially dichotomous set. Although we often see terms such as Highly Satisfactory or Moderately Satisfactory added to make a full scale, the underlying quality (satisfaction) is essentially dichotomous - you are either satisfied or you are not. The following scale, although not generally used owing to the lack of such terms as 'satisfactory' or 'excellent' which are popular buzzwords in our culture, DOES meet the conditions of parallel, balanced responses which could be used in the evaluation of performance:
(++) Very Good
(+) Good
(- ) Poor
(-- ) Very Poor
We could balance this scale in the following way if we wished. The scale responses would not be completely parallel but it might be a little more politically acceptable since it now contains the terms "Excellent" and "Unsatisfactory".
Excellent (+++)
Very Good (++)
Good (+)
Poor (- )
Very Poor (-- )
Completely Unsatisfactory (--- )
4. Use the "OMIT" or "LEAVE BLANK" option for recording "Don't Know" or "Not Applicable" responses.
A common error in constructing questionnaires or rating forms is to include a response position such as "Not Applicable" or "Don't Know" as part of the response scale. The danger here is that such a response position may take on a value (usually 0) when the response statistics are computed and can lead to inaccurate data and poor or erroneous interpretations. The best approach is to simply instruct the respondent to leave blank or otherwise omit responding to any item for which they do not have an answer.
On occasion, however, 'don't know' or 'not applicable' responses have definite meaning and provide important information to the person constructing the scale. In such cases make sure these response choices meet the same criteria for inclusion in a scale as noted in items 1 - 3 above.
|
