Post navigation

Syllabus Rubric

 

Overview
To enrich the resources for measuring the impact of educational development work, we have created a rubric to assess the degree to which a syllabus achieves a learning orientation. The rubric provides qualitative descriptions of components that distinguish learning-focused syllabi and uses a quantitative scoring system that places syllabi on a spectrum from content-focused to learning-focused. It is flexible enough to accommodate a diverse range of levels, disciplines, institutions, and learning environments yet nuanced enough to provide summative information to developers using the tool for assessment purposes and formative feedback to instructors interested in gauging the focus of their syllabi.

Our syllabus rubric was awarded the 2014 POD Network Innovation Award.  Here’s the project summary created for the annual conference in Dallas, TX: POD Innovation Award Poster.

User Guide
To help others use the rubric effectively, we’ve created a User Guide that includes 1) relevant background information and details about validity and reliability, 2) the complete rubric, scoring directions, and a blank scoring sheet, 3) references, 4) and six reference syllabi which we’ve scored and fully annotated. Because this is a living document, check back regularly so that you have the most up-to-date version.

You may use our syllabus rubric and associated resources for research purposes provided you cite the following:

We have scored and fully annotated six different syllabi to help users of our syllabus rubric understand the meaning of low, moderate, and strong evidence.  For the norming process, we recommend users score the six reference syllabi (Appendix C) first without the aid of our scores and annotations. Then, compare scores, referring to the annotations when discrepancies exist (Appendix D).

Analysis Data from UVa’s Course Design Institute
Using a one-group pretest-posttest study design, we have analyzed 54 pre-/post-CDI syllabi pairs. Figures 1-2 show pre- and post-CDI syllabi scores, respectively. Total pre-CDI scores range from 0 (2 syllabi) to 46 points (1 syllabus). The mean score was 9.4 (SD=10.0).  Forty-seven syllabi fell in the content-focused range (87%), 5 in the transitional range (9%), and 2 in the learning-focused range (4%).

Figure 1. Pre-CDI syllabi scores (n=54), sorted by overall score. Component-level scores are indicated by color: blue=Learning Goals & Objectives; red=Assessment Activities; purple=Schedule; orange=Learning Environment.

Figure 1. Pre-CDI syllabi scores (n=54), sorted by overall score. Component-level scores are indicated by color: blue=Learning Goals & Objectives; red=Assessment Activities; purple=Schedule; orange=Learning Environment.

Total post-CDI scores range from 12.5 to 46 points (1 syllabus). The mean score was 31.4 (SD=9.32).  Three syllabi fell in the content-focused range (6%), 21 in the transitional range (39%), and 30 in the learning-focused range (55%).

Figure 2. Post-CDI syllabi scores (n=54), sorted by overall score. Component-level scores are indicated by color: blue=Learning Goals & Objectives; red=Assessment Activities; purple=Schedule; orange=Learning Environment.

Figure 2. Post-CDI syllabi scores (n=54), sorted by overall score. Component-level scores are indicated by color: blue=Learning Goals & Objectives; red=Assessment Activities; purple=Schedule; orange=Learning Environment.

The plot of percent gain versus percent pre-CDI syllabus score is shown in figure 3. Using our definitions of low, moderate, and high gain, six instructors fell in the low-gain, 27 in the moderate-gain, and 20 in the high-gain region. The overall percent normalized gain (i.e. the average of the instructors’ percent normalized gains; %<<g>>) was determined to be 60.4% (SD=22.4%).  This indicates that the average instructor is expected to gain 60.4% of the points possible to them regardless of their pre-CDI syllabus scores. In other words, the CDI intervention appears effective at moving the average instructor to at least the transitional range of the rubric.

Figure 3. Plot of % pre-CDI syllabus score vs. % gain. Low-gain (% ≤ 30%), moderate gain (30 30%) regions are separate by solid red and yellow lines. Content-, transitional-, and learning-focused regions for pre-CDI syllabi are separated by dashed vertical lines. The average normalized gain %> is represented by a solid green line.

Figure 3. Plot of % pre-CDI syllabus score vs. % gain. Low-gain (%<g> ≤ 30%), moderate gain (30 < %<g> < 70%), and high-gain (%<g> 30%) regions are separate by solid red and yellow lines. Content-, transitional-, and learning-focused regions for pre-CDI syllabi are separated by dashed vertical lines. The average normalized gain %<<g>> is represented by a solid green line.

As shown in Table 1, statistical significance was found for overall and criterion-level pre- and post-CDI scores. Effect sizes were measured with Pearson’s r (Sullivan & Feinn, 2012), and we found a practical significance (r: 0.2 = small, 0.5 = medium, and 0.8 = large) for all but Schedule.

Table 1. Overall and criterion-level pre- and post-CDI statistics.
Median pre Median post z p r
 Overall 6.50  31.75  -6.39  <.001  -0.61
 Goals & Objectives  0.00  11.25  -6.07  <.001  -0.58
 Assessment Activities  3.25  12.00  -6.17  <.001  -0.59
 Learning Environment  1.00  9.00  -6.37  <.001  -0.61
 Schedule  0.00  3.00  -2.95  .003  -0.28

Questions? Contact Michael Palmer at mpalmer@virginia.edu.