From the Fall of 1995 through the Fall of 1999, the ChemConnections evaluation teams have conducted numerous types of studies in a wide variety of institutions (see Table 1).

As modules were refined, a second round of interviews was conducted to clarify the nature and relative importance of learning issues raised for students in different institutional contexts and class sizes by the experience of well-developed, well-taught modules. By comparison with first-round findings, we could also discern changes over time in student responses to modular learning. Again, individual and focus group interviews were conducted, this time with 323 "modular" students at 10 institutions where beta-version modules were being taught by seasoned modular teachers. The classes again represented a range of institutional types and also variety in how modules were used: 3 classes experienced an all-modular year, 6 an all-modular semester, and 1 a single module as a capstone.

We are also interested in how teachers and institutions change. Data were gathered by recorded live and telephone interviews with individuals and a few small groups, and solicited electronic accounts of developers and adapters’ observations on module teaching. Participants represented these groups:

Analysis of ethnographic text data is time-consuming and cannot be done on partial data sets. However, the following analyses are well-advanced and will be among the first studies published.

Further analyses are underway, including analysis of the 2^nd-round interview data for courses using beta modules and taught by seasoned modular teachers. Several publications are in preparation.

1. In the early classroom use of modules (round 1), students showed gains in the following areas:

2. Early classroom use of modules also highlighted some areas of pedagogical difficulty:

3. We learned that it is more relevant and productive to ask students about what they have gained from specific aspects of the class than what they liked or disliked about the class or teacher. This finding led to the development of the SALG instrument, or Student Assessment of Learning Gains.

In particular, analysis of 12,993 student observations from round-1 interviews showed that students gave both positive and negative ratings to specific aspects of the class and to their teacher’s classroom performance. However, total student evaluation of faculty teaching performance was broadly 50% positive, and 50% negative, for both modular and comparative classes. Thus neither group of faculty got a clear picture of the overall perceived utility of their classroom work.

However, in both modular and comparative classes, students gave clear indications about what they themselves had "gained" both from specific aspects of their classes and overall. When all specific gain-related observations were totaled, 55%of the observations referenced "things gained" (for both types of class); 33% (modular) and 32% (comparative) referenced "things not gained;" and 11% (modular) and 13%(comparative) were "mixed" or qualified gains.

4. Certain areas of risk were identified for those teaching with modules:

5. However, faculty benefited from the formation of learning communities:

The early comparative studies were conducted at two institutions, a small college (Grinnell) and a large university (U.C. Berkeley). The experimental design at each school compared students in a course section taught with modules to those in a section that used a textbook and lecture format.

The assessments in both studies included pre/post conceptual tests, pre/post attitudinal surveys, post-test problem-solving interviews, a post-test ACS exam (1996 Brief version), and in-class exams or quizzes.

The modules used in these studies were draft versions that contained errors, did not always employ the final format of modules, and had fewer graphics than the final versions. Moreover, at the time of the studies, few faculty had used modules. One faculty member at Grinnell had never before taught with modules; the Berkeley study marked the first occasion anyone had attempted to employ modules in a large classroom (over 200 students). Because both studies were conducted early in the production and use of modules, the results of these studies must be taken as preliminary. As the modules evolved and faculty use of them matured, the student outcomes undoubtedly changed.

Given these caveats, the results of the early comparative studies are as follows. At both schools, students in the modular section outperformed the control group on conceptual problems in chemistry and on scientific thinking problems. No difference was seen in their performance on the standard ACS exam. Modular section students at the large university also outperformed their peers on the first midterm exam in the subsequent organic chemistry course. See Table 2 for student means on these assessments.

The post-course attitude survey showed that the Modular section students were more positive about chemistry and the course than their peers in the control section at Grinnell College. However, at U.C. Berkeley, the opposite attitudinal pattern was found. See Table 3 for a selected sample of post-course survey questions that showed statistically significant differences at both institutions.

An analysis of informal focus group data and data from a modular course held the subsequent semester suggests that much of the negative response at U.C. Berkeley was due to the embryonic nature of the course; modules had never before been employed in a large course. In the subsequent semester, students’ attitudes became more positive, roughly matching attitudes held by students in the control group section (Table 3).

A strong effect of Graduate Student Instructor (teaching assistant) was also found on the attitude of students in the modular section. Students’ attitudes toward the modular course paralleled those of the GSI for their section, though GSI and student attitudes for the non-modular section were uncorrelated. This shows the importance of good training for all teaching staff when new instructional methods are introduced. A separate study underway at Berkeley is designed to address the issue of TA professional development.

* Modular vs. non-modular difference within institution is statistically significant at p<0.05 level. Differences between sections at institutions are not comparable.

‡ All written assessments employed SAT scores as a covariate in the ANOVA to remove the effect of differences in SAT scores.

‡‡ All written assessments were analyzed using a General Linear Model with SAT, Section (Modular or Non-Modular) and Graduate Student Instructor (nested within Section) as independent variables to remove effect of difference in SAT and GSI .

Note: All group differences within an institution are significant at the 0.05 level. Differences between institutions are not comparable.