Evaluation

 

Process

Types of rubrics: Analytic scales

Analytic scales are usually associated with generic rubrics and tend to focus on broad dimensions of writing or speaking performance. These dimensions may be the same as those found in a generic, holistic scale, but they are presented in separate categories and rated individually. Points may be assigned for performance on each of the dimensions and a total score calculated.

Traditionally, analytic rubrics are associated with large-scale assessment of general dimensions of language performance. However, analytic rubrics certainly can be created or adapted for use in classroom settings and with particular tasks (e.g., Taggart et al., 1998). These rubrics often combine performance categories from a generic rubric with categories directly related to a task, such as demonstrating understanding of specific lesson content (Moskal, 2000). In practice, the names "analytic rubric" and "multiple trait rubric" may be used interchangeably.

Performance dimensions commonly found in analytic rubrics include:

Speaking & Writing

• Content
• Vocabulary
• Accuracy/Grammar/Language Use
• Task fulfillment
• Appropriate use of language
• Creativity
• Sentence structure/Text type
• Comprehensibility

Writing

• Organization
• Style
• Mechanics
• Coherence and Cohesion

Speaking

• Fluency
• Pronunciation
• Intonation

Figure F presents an adaptation of a well-known analytic scale for evaluating ESL writing performance. Describing this rubric, Tedick (2002) writes: "Note that the scale assigns different weights to different features. This allows a teacher to give more emphasis to content than to grammar or mechanics, for example. The option to weigh characteristics on the scale represents an advantage to analytic scoring." (p. 35).

Figure F2 shows an analytic scale for role plays and interviews used with students in first-year French courses at the University of Minnesota. This rubric can be used with other languages. In this example, all criteria are weighted equally.

There are more sample analytic rubrics in the Evaluation > Examples section.

Advantages:

• They provide useful feedback to learners on areas of strength and weakness.
• Their dimensions can be weighted to reflect relative importance.
• They can show learners that they have made progress over time in some or all dimensions when the same rubric categories are used repeatedly (Moskal, 2000).

Disadvantages:

• "The whole is greater than the sum of its parts." Tedick (2002) notes: "Separate scores for different aspects of a student’s writing or speaking performance may be considered artificial in that it does not give the teacher (or student) a good assessment of the "whole" of a performance." (p. 36).
• They take more time to create and use.
• There are more possibilities for raters to disagree. It is more difficult to achieve intra- and inter-rater reliability on all of the dimensions in an analytic rubric than on a single score yielded by a holistic rubric.
• There is some evidence that raters tend to evaluate grammar-related categories more harshly than they do other categories (McNamara, 1996), thereby overemphasizing the role of accuracy in providing a profile of learners' proficiency.
• There is some evidence that "when raters are asked to make multiple judgement, they really make one..." (Fulcher, 2009). Care must be taken to avoid a "halo effect" and focus on the individual criteria to assure that diverse information about the learner's performance is not lost.