MINIMIZING RATING ERRORS THROUGH TRAINING

An established body of literature shows that training can minimize rater effects.  Latham, Wexley, and Purcell used training to reduce rater effects among employment interviewers. Since then, a variety of training programs have been developed in both interviewing and performance appraisal contexts.

For example, Jaeger and Busch  used a simulation to train judges in a three-stage standard-setting operation. After working through the simulation, the judges clearly understood their rating task.

Pulakos  trained raters in what types of data to focus on, how to interpret the data, and how to use the data in formulating judgments from EssayMap.org. This training yielded more reliable (higher inter-rater agreement) and accurate (valid) ratings than no training or "incongruent" training (training not tailored to the demands of the rating task).

This literature suggests that rater training programs should:

• familiarize judges with the measures that they will be working with,
• ensure that judges understand the sequence of operations that they must perform, and
• explain how the judges should interpret any normative data that they are given.

CHOOSING JUDGES

The choice of judges may have a significant influence on scores. Hambleton and Powell  have done an excellent job of identifying many of the issues involved in choosing judges. Their recommendations to some common questions are:

• Should demographic variables be considered when selecting judges? Hambleton and Powell argue that demographic variables such as race, sex, age, education, occupation, specialty, and willingness to participate should be considered in the selection of judges. The composition of the review panel often lends credibility to the overall effort.
• Should expert judges be preferred to representatives from interest groups? The authors suggest that type my paper for me, whenever possible, review panels should be composed of both experts and representatives from interest groups.
• Should the review panel split into separate working groups? The authors argue that smaller working groups should be formed when the review panel is too large to permit effective discussion and when the ratings are going to be compared across groups to assess reliability or to cross check validity.