This article describes how we at the Office of Measurement Services collect enrollment information for the Student Rating of Teaching (SRT), and how we compute response rate for each course. It will enumerate some of the challenges associated with that task, and solutions in place to generate the most accurate report possible. It will also explore some of the ways that this process can lead to counterintuitive results.

Challenges
When preparing a report of the SRT response rate for a course, there are two necessary pieces of information. First, we need the official enrollment, obtained from the central PeopleSoft system. Second, we must count the completed SRT response sheets that were submitted for that course.

Unfortunately, the official enrollment for a given course is not necessarily sufficient, for a number of reasons.

First, it should be noted that, in order to meet deadlines and internal standards of data integrity, it is necessary for OMS to work with a copy of the PeopleSoft data. This data is copied from PeopleSoft as late as possible, to ensure that the most current data is available. However, this data must be copied before the half-semester courses are evaluated. So, when this document refers to data in the central system, it refers to the data as it stood prior to midterm. Thus, some discrepancies may be explained by the changes in enrollment between the time the data is copied, and the actual time of administration.

In some instances, the enrollment information in the central system might be outdated or missing. This is especially the case for some of the less-traditional classes, like College in the Schools, courses from the College of Continuing Education, any of the programs abroad, or courses otherwise not having a classroom on campus.

There are also real-world logistical challenges that can lead to further discrepancies. For instance: often, two or more courses will be "cross-listed", implying that they meet in the same room at the same time. When the in-class evaluations are collected, the procedure indicates that these two classes' sheets are to be separated into different batches. However, this is often not done correctly, and since the sheets are anonymous, there is no feasible way to separate them correctly after the fact.

Another example of real world challenges includes students that, for one reason or another, are not technically enrolled in the course, and yet submit an in-class evaluation. This could include auditing students, or guests, or students whose enrollment simply hasn't been properly registered in the central system.

While by no means exhaustive, these examples give a flavor for the differences that can exist between the official enrollment reported in PeopleSoft, and the actual number of students participating in the class and evaluating the instruction.

Solutions, single courses
When addressing these challenges, we focused on the primary use of these statistics. That is, the enrollment is used as a baseline from which to compute response rate. This is the main reason we have to collect enrollment data. So, our answers to the challenges listed above should be made with that end result in mind. What solution would give the most reasonable response rate, given the information we have? Also, for data interpretation reasons, it is important that the computed response rate be no larger than 100 percent, so any solution must also make that guarantee as well.

Another thing to note is that, due to the volume of course evaluation data that is processed and the required limits on processing time, all solutions must be completely automatic. That is, the software used to generate reports from the raw data must be able to implement the solution without individual intervention from a person.

First, we simply assume that, if everything seems right, then it is right. That is, if the number of sheets we received was less than or equal to the reported official enrollment for a course, then that is enough information to compute a response rate, and that is the response rate that is used.

Second, if we received more sheets than the official enrollment would account for, we check to see if the course was cross-listed with any others. The assumption we make is that, for this course, sheets from the other, cross-listed course or courses were mistakenly included in this course's batch. If we find that this might be the case, then we must assume that all the evaluations from that room (with all the cross-listed courses' students in attendance) were sent in together. So, the response rate should be computed using the sum of all the cross-listed courses' official enrollments.

If we do report this as the enrollment, the report indicates this fact. It also lists the courses whose enrollments were combined to make the reported number.

Third, if the cross-listed courses' enrollments were still not enough to account for all the evaluations sent in for the course, the report indicates that the response rate is 100 percent. Thus, we meet the requirement that no response rate is greater than 100 percent, but we make no further assumptions that we might know what the "official" enrollment might be. Note that the "official" enrollment is reported as it was found in the PeopleSoft system; only the response rate is changed.

Again, if this enrollment figure is used, the report is marked to indicate this fact.

Solutions, aggregate reports
For reports including aggregated date for multiple courses, the situation is much the same. We count the received evaluation sheets for all the courses in question, and we look up the official enrollment for each. If the number of sheets is less than the sum of the official enrollments, we use that information to compute a response rate.

If there are more sheets than official enrollments would account for, we then pull together all the courses that are cross-listed with any of the courses included in the aggregate. All of those official enrollments are summed, and used as the baseline.

If there are still more sheets than the cross-listed enrollments would account for, the response rate is declared to be 100 percent, and the original "official" enrollment is reported, as described above.

Sources of Counterintuitive Results
When working with these reports, the above solutions may result in some counterintuitive results. Specifically, when comparing enrollment totals between a large aggregate, and a set of smaller sub-aggregates, some enrollment totals may not align. Three examples of this are listed and explained below.

Example 1
The first example is that if some small reports (e.g. individual courses or small aggregates) have a small number of response sheets, they will not be reported. This is the "less than five" rule, which prevents a report containing less than 5 student responses from being published, to protect the students' anonymity. However, in the larger aggregate report, these responses could be included, since they do not endanger the students' anonymity in the larger aggregate. Thus, the total number of response sheets reported will contain a difference. (While this is possible, currently we do not include any courses with fewer than five responses in aggregate totals unless a specific request is received to do so and it is determined to be a valid request.)

Example 2
The second example is somewhat more subtle, and involves the doubling of cross listed courses. Consider the case where there are three courses cross-listed together, and we generate an aggregate report that contains two of those classes. Suppose that, because of the number of sheets received for this aggregate, we must use the cross-listed enrollments, and so we sum the enrollments for the three cross-listed courses to include in our "official" enrollment.

Now, we create two smaller aggregates, each containing one of the three cross-listed courses. Again, each smaller aggregate must use the sum of all three cross-listed courses in their tally of "official" enrollment. Note, however, that the smaller aggregates, when observed side-by-side, are counting the enrollment for each cross-listed course multiple times. The larger aggregate only counts the enrollment for each course once. So, the sum of the enrollments of the smaller aggregates will not be the same as the enrollment for the larger aggregate. This discrepancy will be seen in a report with multiple layers of subgrouping.

Example 3
Finally, it is worth pointing out that sometimes confusion will arise when the number of sheets is greater than the "official" enrollment, and the cross-listed courses do not adequately explain the larger numbers. As described above, the number presented is still the "official" enrollment straight from PeopleSoft. However, if this is simply a data report, not an officially rendered SRT Report, it may be that no explanatory note is present. In this case, direct computation of response rate using the number of sheets and this enrollment will yield a number greater than 100 percent. The SRT Report rendering system manually adjusts the response rate to 100 percent, and it is necessary for the report generator to do the same.