Journal of Engineering Education, 87(4), 469-480 (1998).

A LONGITUDINAL STUDY OF ENGINEERING
STUDENT PERFORMANCE AND RETENTION.
V. COMPARISONS WITH
TRADITIONALLY-TAUGHT STUDENTS

Richard M. Felder
Department of Chemical Engineering
North Carolina State University

Gary N. Felder
Department of Physics
Stanford University

E. Jacquelin Dietz
Department of Statistics
North Carolina State University


Abstract

In a longitudinal study at North Carolina State University, a cohort of students took five chemical engineering courses taught by the same instructor in five consecutive semesters. The course instruction made extensive use of active and cooperative learning and a variety of other techniques designed to address a broad spectrum of learning styles. Previous reports on the study summarized the instructional methods used in the experimental course sequence, described the performance of the cohort in the introductory chemical engineering course, and examined performance and attitude differences between students from rural and urban backgrounds and between male and female students.1-4 This paper compares outcomes for the experimental cohort with outcomes for students in a traditionally-taught comparison group. The experimental group outperformed the comparison group on a number of measures, including retention and graduation in chemical engineering, and many more of the graduates in this group chose to pursue advanced study in the field. Since the experimental instructional model did not require small classes (the smallest of the experimental classes had 90 students) or specially equipped classrooms, it should be adaptable to any engineering curriculum at any institution.

I. Introduction

Shrinking engineering enrollments pose a potentially serious problem for American industry and society. The annual graduation rate in engineering has decreased by roughly 20 percent in the last decade, while a rise of 25-30% in engineering jobs by the end of the century has been predicted.5 Both the increasing difficulty of attracting high school graduates into engineering and high attrition rates of enrolled engineering students have contributed to the decline in the graduation rate. Most engineering schools have undertaken major recruitment efforts to address the first problem, many of them directed at women and minorities. These efforts have limited potentials for success, however, since freshman enrollment is heavily influenced by factors out of the university's control, including fluctuations in the job market and starting salary levels in engineering relative to other fields.6 Engineering schools seeking to improve their graduation rates are therefore starting to see improving retention as the most effective way to do it.

Considering the strong academic records of most students who choose to go into engineering, rates of attrition from the curriculum are dramatic. In his monumental study of nearly 25,000 students at over 300 institutions, Astin7-9 found that only 43% of the first-year engineering students in his population went on to graduate in engineering. Moller-Wong and Eide10 obtained similar results for a cohort of 1,151 engineering enrollees at Iowa State University. They found that after five years, 32% of their subjects graduated in engineering and 13% were still enrolled, for a probable eventual graduation rate between 40% and 45%.

A common but incorrect explanation of high student attrition rates from engineering is that most of those who leave lack the academic ability to cope with the rigors of the discipline. In fact, studies have shown little difference in academic status between students staying in engineering and students leaving.11-13 The true explanation appears to involve a complex set of factors including students' attitudes toward engineering, their self-confidence levels, and the quality of their interactions with instructors and peers,7,11,14,15 along with their aptitude for engineering.

In turn, students' attitudes toward engineering and confidence levels are strongly related to their classroom experience. Astin7-9 shows that compared to majors in other fields, engineering majors are much more dissatisfied with the quality of instruction they receive in college and with their overall college experience, and he offers convincing evidence that the prevalent model of instruction in engineering-extensive reliance on lecturing and individual work and norm-referenced grading (curving)-plays a major role in this high dissatisfaction level and therefore in student attrition.

Astin, Cross, and many other educational scholars recommend establishing an alternative instructional environment that includes using active and cooperative learning and a variety of other pedagogical methods designed to accommodate different learning styles.15-20 Although thousands of research studies attest to the educational benefits of such an environment, most reported results are for tests in a single experimental course. The longitudinal study described in this paper was designed to use a number of proven instructional methods in a sequence of chemical engineering courses, and to determine how repeated exposure to the methods in a natural classroom setting would affect the students' performance and retention.

Previous reports on the study summarized the instructional methods used in the experimental course sequence, described the performance of the cohort in the introductory chemical engineering course, and examined differences in performance and attitudes between students from rural and urban backgrounds and between male and female students.1-4 This paper compares outcomes for the experimental cohort with outcomes for students in a traditionally-taught comparison group. We first review the design of the study and present demographic and academic profiles of the experimental and control groups, following which we summarize the between-group comparison data and discuss the implications of the outcomes for engineering education.

II. Experimental Design

The subjects of the longitudinal study, henceforth designated the experimental group, are 123 students who enrolled in the introductory chemical engineering course in the Fall 1990 semester. The students in this group who remained in sequence in the chemical engineering curriculum were taught by one of us (RMF) in the following five courses:

  1. CHE 205 - Chemical Process Principles. Material and energy balances on chemical processes, basic concepts and calculations.
  2. CHE 225 - Chemical Process Systems. Process variable measurement methods, computer simulation of processes, applied statistical analysis.
  3. CHE 311 - Transport Processes I. Fluid dynamics and heat transfer.
  4. CHE 312 - Transport Processes II. Mass transfer and separation processes.
  5. CHE 446 - Chemical Reactor Design and Analysis.

The instruction in the experimental course sequence included extensive active and cooperative learning, open-ended questioning, multidisciplinary problem formulation and solution exercises, criterion-referenced grading, and other features designed to address a full spectrum of student learning styles.16 No unusual classroom facilities were provided. The class size varied from 90 to 123 students, the classroom was a fixed-seat auditorium, the chalkboard was the only medium of instruction except for the occasional use of an overhead projector, and no in-class computing facilities were available. The approach should therefore be accessible to any instructor in any educational setting.

Background information on the experimental group was collected in the first course, including demographic data, SAT scores, first-year grade-point averages and grades in selected first-year courses, and scores on both the Myers-Briggs Type IndicatorÒ (MBTI), a personality inventory based on Jung's theory of psychological types, and the Learning and Study Strategies InventoryÒ (LASSI), an instrument that assesses students' test-taking skills and strategies, motivation to learn, and anxiety levels. Subsequently collected data included course grades, statistics on persistence in the chemical engineering curriculum, and students' self-evaluations and reactions to their educational experiences.

The same data were collected for 189 students enrolled in CHE 205 in the Fall 1992 semester. This comparison group proceeded through the curriculum taught only by instructors who used traditional methods. Our hypotheses were that the experimental group would have a higher retention in chemical engineering, more positive attitudes toward their instruction, and greater confidence levels in their problem-solving skills.

Tables 1-11 report data for the students in the experimental and comparison groups. The responses shown for each item almost always fall short of the total populations of the two groups, for several reasons. Some students failed to respond to particular questionnaire items; some dropped out of the five-semester course sequence after failing a course, entering the co-op program, or transferring out of chemical engineering; and some were in other curricula like Pulp and Paper Technology that only required the first chemical engineering course in their degree programs. All reported levels of significance are derived from two-tailed Fisher's exact tests unless otherwise noted. "Statistically significant" denotes p<.1; statistically significant p values are marked in tables by asterisks.

III. Profiles of the Experimental and Comparison Groups

Table 1 shows demographic data for the two groups of students. The ethnic distributions and the split between rural/small town backgrounds and urban/suburban backgrounds were roughly the same for both groups. The comparison group had a higher percentage of women enrolled, although the difference was not statistically significant. The only significant difference was that a greater percentage of students in the experimental group had parents trained in science.

TABLE 1
DEMOGRAPHIC DATA

 

Expt

Comp

p

Home community
Rural
Small town
Suburban
Urban

(N=120)
14%
32%
37%
18%

(N=176)
18%
34%
29%
19%

 

.57

Ethnic background
African-American
Caucasian
Asian-American
Other

(N=120)
6%
82%
5%
7%

(N=175)
7%
81%
10%
2%

 

.12

Gender
Male
Female

(N=123)
71%
29%

(N=189)
64%
36%


.17

Parents trained in science
Both
Mother
Father
Neither

(N=62)
24%
5%
40%
31%

(N=78)
12%
4%
27%
58%

 

.01*

Table 2 shows the precollege academic credentials of the two groups. The experimental group had slightly higher SAT's and essentially identical AI's (Admissions Index, a predicted grade point average based on SAT scores and high school performance data). The comparison group had more advanced placement credits for first-year core courses (the first and second calculus courses, the first and second general chemistry courses, the introductory physics course, and freshman English), with the difference being significant in the introductory chemistry course (CH 101). Table 3 shows average grades for the two groups in the first-year core courses. The experimental group did much better in the second general chemistry course (CH 107), but the difference between the overall first-year grade-point averages recorded by the two groups was negligible. In short, the experimental and comparison groups entered the CHE curriculum with very similar academic credentials.

TABLE 2
PRECOLLEGE ACADEMIC CREDENTIALS

 

Expt

Comp

p

Admissions criteria

SATM
SATV
Admissions Index (PREGPA)

(N=101)
629
522
2.87

(N=147)
613
510
2.86


.15
.24
.89

Percentage receiving AP credits in courses

MA141
MA241
CH101
CH107
PY205
ENG111

(N=116)
19%
3%
3%
3%
2%
47%

(N=172)
20%
1%
9%
6%
5%
47%


.88
.22
.09*
.42
.32
.90

t-test

     

TABLE 3
FIRST YEAR ACADEMIC PERFORMANCE

 

Expt (N)

Comp (N)

p

First year course grades
    MA141
    MA241
    CH101
    CH 107
    PY205
    ENG111

 
3.30(103)
3.00(106)
3.46(105)
3.30(105)
2.74(102)
4.02 (99)

 
3.55(141)
2.93(147)
3.44(146)
2.83(147)
2.85(152)
4.14(135)

 
.24
.89
.58
.002*
.70
.49

Overall GPA

3.24 (100)

3.20(139)

.39

A=4.0, AP credit=5.0
Wilcoxon's rank-sum test

The two groups also had similar profiles on both the Myers-Briggs Type Indicator and the Learning and Study Strategies Inventory. On the MBTI, experimental and comparison groups both showed preferences for extraversion over introversion, sensing over intuition, thinking over feeling, and judging over perceiving. None of the between-group preference differences was statistically significant. The scores on all 10 dimensions of the LASSI were almost identical for both groups.

IV. Between-Group Comparison Data

Tables 4-11 summarize differences between the experimental and comparison groups in course grades, attitudes, self-confidence levels, and rates of graduation and attrition from chemical engineering after five years of college. Also shown are the seniors' career goals, self-ratings of changes in selected abilities since they enrolled in CHE 205, and ratings of their education. In this section we simply present the results, and in the Discussion section we draw inferences from them.

A. Performance in Experimental Sequence Courses

Table 4 shows grades earned by students in both groups in the five courses that constituted the experimental sequence.

TABLE 4
GRADES IN EXPERIMENTAL SEQUENCE COURSES

 

205

225

311

 

Expt

Comp

p

Expt

Comp

p

Expt

Comp

p

N

123

189

 

71

102

 

67

93

 

Grade
    A
    B
    C
    D
    F/Drop

 
23%
33%
12%
8%
24%

 
12%
23%
46%
5%
15%

 
 
 
<.001*

 
39%
39%
15%
3%
3%

 
22%
36%
33%
7%
2%

 
 
 
<.02*

 
28%
31%
30%
6%
4%

 
25%
25%
41%
6%
3%

 
 
 
<.66

% A

23%

12%

.01*

39%

22%

.02*

28%

25%

.72

% Pass

68%

80%

.02*

94%

91%

.56

90%

90%

1.00

Avg. grade (A=4)

2.23

2.12

.11

3.10

2.69

.002*

2.73

2.61

.37

 

 

312

446

 

 

Expt

Comp

p

Expt

Comp

p

N

61

74

 

54

52

 

Grade
    A
    B
    C
    D
    F/Drop

 
43%
38%
20%
0%
0%

 
20%
34%
36%
5%
4%

 
 
 
<.004*

 
31%
46%
22%
0%
0%

 
29%
44%
17%
6%
4%

 
 
 
<.02*

% A

43%

20%

.008*

31%

29%

.83

% Pass

100%

91%

.02*

100%

90%

.03*

Avg. grade (A=4)

3.23

2.61

.001*

3.09

2.88

.23

"Passing" denotes C or better, required to advance in the curriculum
Wilcoxon's rank sum test

The grade distribution in the experimental course offering of CHE 205 was bimodal, with 56% of the students earning A's and B's, 12% earning C's, and 32% earning D's and F's or dropping the course. (Receiving either D or F meant that the students had to repeat the course if they wished to continue in the chemical engineering curriculum.) The distribution in the comparison offering was unimodal, with 35% A's and B's, 46% C's, and 20% D's, F's, and drops.

The latter distribution-many C's, a smaller number of A's and B's, and a moderate number of failures and dropouts-is somewhat unusual for traditionally-taught offerings of this course, which are characterized by much higher failure rates. The reason that only 20% "failed" (D or lower) was that a radical curving system was used to assign grades. Arbitrary break points in the weighted average numerical grade distribution were used as boundaries between letter grades, and numerical grades ranging from the high 40's to the low 70's qualified for a letter grade of C.

In courses after CHE 205, the grades in the experimental offerings were consistently better than those in the comparison offerings. In the fourth and fifth courses in the sequence, no students in the experimental group and roughly 10% of those in the comparison group received D's and F's.

B. Performance in Other Chemical Engineering Courses

Besides the five courses that constituted the experimental sequence, the chemical engineering curriculum includes two thermodynamics courses taken in the junior year, two laboratory courses, lecture courses on process design and process control taken in the senior year, and a project-based senior design course. None of these courses was taught by Dr. Felder to either group of students.

The grades in the laboratory and senior lecture and design courses tend to be high and provide little basis for comparison between the two student groups, but the junior thermodynamics courses (CHE 315 and CHE 316) are considered by many to be the hardest courses in the curriculum. Table 5 shows the distributions of grades received in these courses by both groups. The average experimental group grade was half a letter grade above the average comparison group grade, and relative to the comparison group, the experimental group earned almost twice the percentage of A's and less than half the percentage of D's and F's. The between-group performance differences were all highly significant.

TABLE 5
GRADES IN OTHER CHEMICAL ENGINEERING COURSES

 

315/316

Expt

Comp

p

N (Number of grades)

167

202

 

Grade
    A
    B
    C
    D
    F


29%
40%
26%
3%
3%


16%
34%
35%
10%
5%

 
 
 
.002*

% A

29%

16%

.006*

% Pass

100%

91%

.02*

Avg. grade (A=4)

3.23

2.61

.001*

"Passing" denotes C or better, required to advance in the curriculum
Wilcoxon's rank sum test

C. Ratings of the Chemical Engineering Courses

At several points in the study the students were asked to rate their preparation for their current course by its prerequisite courses. The results are shown in Table 6. The comparison group rated its preparation for CHE 205 slightly higher than the experimental group did, although the difference was not significant. Thereafter, however, the experimental group ratings were dramatically higher, with the differences significant at the .001 level. The experimental group students also consistently considered their courses more interesting (Table 7), with the differences being statistically significant when the questions referred specifically to the courses in the experimental sequence.

TABLE 6
RATING OF PREPARATION PROVIDED
BY PREREQUISITE COURSES

 

Expt

Comp

p

How well prepared for 205?
    Unusually weak
    Less than average
    About average
    More than average
    Very strong

(N=118)
1%
10%
50%
36%
3%

(N=174)
3%
9%
45%
32%
10%

 
 
 
.16

How well prepared for 311 by 205 and 225?
    Poor
    Fair
    Average
    Good
    Excellent

(N=63)
0%
3%
10%
43%
44%

(N=78)
9%
15%
40%
32%
4%

 
 
 
<.001*

How well prepared for 312 by 311?
    Poor
    Fair
    Average
    Good
    Excellent

(N=60)
3%
3%
15%
42%
37%

(N=75)
7%
21%
43%
28%
1%


 

<.001*

How well prepared for senior design?
    Don't know
    Poor
    Fair
    Good
    Excellent

(N=63)
10%
2%
11%
59%
19%

(N=82)
7%
1%
32%
51%
9%


 
 
<.001*


TABLE 7
INTEREST LEVEL OF COURSES
RELATIVE TO EXPECTATIONS

 

Expt

Comp

p

311
    Much more
    Somewhat more
    What I expected
    Somewhat less
    Much less

(N=63)
10%
46%
35%
35%
8%

(N=78)
3%
31%
46%
46%
15%

 
 

.06*

312
    Much more
    Somewhat more
    What I expected
    Somewhat less
    Much less

(N=60)
5%
45%
43%
5%
2%

(N=75)
1%
16%
53%
21%
8%

 
 

<.001*

446
    Much more
    Somewhat more
    What I expected
    Somewhat less
    Much less

(N=61)
23%
33%
39%
3%
2%

(N=31)
3%
32%
52%
13%
0%

 
 

.03*

All CHE courses (seniors)
    Much more
    Somewhat more
    What I expected
    Somewhat less
    Much less

(N=66)
9%
55%
24%
11%
2%

(N=82)
9%
45%
29%
13%
4%

 
 

.79

Table 8 shows the students' perceptions of their workload and their estimates of average time they spent outside class each week on several of the experimental sequence courses and (as seniors) on all of their chemical engineering courses. The two groups were divided on the two junior courses. More in the comparison group found the workload in CHE 311 unreasonable, although the two groups spent comparable time outside class on this course. On the other hand, the experimental group spent far more time on CHE 312 and many more of them found the workload in 312 unreasonable relative to the comparison group. The seniors in both groups gave identical ratings to the average workload for all chemical engineering courses, and the reported time spent outside class was not significantly different for the two groups.

TABLE 8
PERCEPTION OF COURSE WORKLOADS

 

Expt

Comp

p

Workload in 311
    Reasonable
    Too heavy
    Much too heavy

(N=63)
71%
17%
5%

(N=78)
40%
55%
5%


 
<.001*

Workload in 312
    Reasonable
    Too heavy
    Much too heavy

(N=60)
43%
52%
5%

(N=75)
63%
33%
4%

 

.08*

Workload in CHE courses
    Reasonable
    Too heavy
    Much too heavy

(N=66)
56%
44%
0%

(N=81)
56%
44%
0%


 
1.00

Average hr/wk spent outside class on 311
    <4
    4-8
    9-12
    13-16
    >16

(N=63)
2%
32%
44%
17%
5%

(N=78)
0%
31%
50%
15%
4%

 
 
 
.87

Average hr/wk spent outside class on 312
    <4
    4-8
    9-12
    13-16
    >16

(N=60)
3%
30%
42%
20%
5%

(N=75)
8%
63%
20%
7%
3%

 
 
 
<.001*

Average hr/wk spent outside class on CHE courses (seniors)
    <4
    4-8
    9-12
    13-16
    >16

(N=66)
 
3%
21%
45%
17%
13%

(N=82)
 
3%
27%
34%
23%
14%

 
 
 
 
.65

D. Fifth Year Graduation and Attrition Rates

Chemical engineering is in principle a four-year curriculum at N.C. State, but most students who graduate in this field do so after 4 1/2 or 5 years. Table 9 shows the status of students in both groups four years after enrolling in CHE 205 (which for most of them would be at the end of their fifth year of college). Data are shown only for the students who enrolled in CHE 205 with the intention of obtaining degrees in chemical engineering.

TABLE 9
STATUS AFTER FIVE YEARS OF COLLEGEa

 

Expt
(N=95)

Comp
(N=139)

p

Statusb
    G

 
85%

 
65%

 
<.01* d

    S
    TG
    TB
    DG
    DB

0%
4%
6%
1%
3%

3%
8%
14%
8%
3%

 

Retentionc
    Retained
    Not retained


85%
15%


68%
32%


<.01* d

aData only for intended CHE majors
bG = graduated with CHE degree
  S = still enrolled in CHE
  TG = left CHE in good standing (GPA > 2.00)
  TB = left CHE in bad standing (GPA < 2.00)
  DG = left NCSU in good standing (GPA > 2.00)
  DB = left NCSU in bad standing (GPA < 2.00)
c Retained = status of G or S
  Not retained = status of TG, TB, DG, DB
dchi-square test

The five-year graduation rate in chemical engineering was 85% for the experimental group and 65% for the comparison group, a difference significant at the .01 level. Three percent of the comparison group students and none in the experimental group were still enrolled in chemical engineering after five years of college. Students in the comparison group were more than twice as likely to transfer out of chemical engineering to a different curriculum (22% vs. 10%) and eight times more likely to drop out of college in good academic standing (8% vs. 1%). The same percentage in both groups (3%) dropped out in bad standing.

E. Seniors' Post-graduation Plans, Concerns, and Self-Ratings of Abilities

Seniors in the experimental group were more than twice as likely as comparison group students to express an intention of pursuing graduate study in chemical engineering (18% vs. 7%) and almost twice as likely to want to work at either at a university or a research facility (15% vs. 8%). The percentage of the comparison group expressing very high anxiety about being a chemical engineer following graduation was almost triple the percentage of the experimental group doing so (17% vs. 6%).

The seniors were asked to rate the changes in certain of their abilities since the beginning of CHE 205 (Table 10). The experimental group students gave themselves higher ratings for improvements in both basic problem-solving ability and creative problem-solving ability, with the latter difference being statistically significant. The comparison group students gave themselves significantly higher ratings for improvements in ability to solve computer problems and to work independently. There was almost no difference in the ratings of improvement in ability to work in a group.

TABLE 10
SENIORS' RATING OF CHANGES IN
ABILITIES SINCE START OF CHE 205

 

Expt

Comp

p

Change in ability to solve basic engineering problems
    Much higher
    Higher
    About the same
    Lower
    Much lower

(N=66)
50%
36%
8%
5%
2%

(N=81)
38%
53%
7%
1%
0%


 

.16

Change in ability to solve creative engineering problems
    Much higher
    Higher
    About the same
    Lower
    Much lower

(N=66)
 
21%
64%
9%
6%
0%

(N=81)
 
12%
58%
25%
4%
1%


 
 

.06*

Change in ability to work in a group
    Much higher
    Higher
    About the same
    Lower
    Much lower

(N=66)
27%
50%
20%
2%
2%

(N=80)
24%
51%
25%
0%
0%


 

.60

Change in ability to solve computer problems
    Much higher
    Higher
    About the same
    Lower
    Much lower

(N=66)
35%
44%
21%
0%
0%

(N=81)
46%
46%
7%
1%
0%


 

.06*

Change in ability to work independently
    Much higher
    Higher
    About the same
    Lower
    Much lower

(N=66)
5%
32%
52%
9%
3%

(N=80)
11%
52%
29%
8%
0%


 

.009*

F. Seniors' Ratings of Their Education

Table 11 shows the seniors' ratings of the quality of their course instruction, the student-friendliness of the instructional environment in the chemical engineering department, the support they got from their peers (classmates), and the value of their investment in their chemical engineering education. In all cases, the ratings from the experimental group students were significantly higher than those for the comparison group students, with the levels of significance for the first three categories being less than .001. The percentage of the experimental group giving their instruction the top rating was an order of magnitude greater than the percentage for the comparison group.

TABLE 11
SENIORS' RATINGS OF CHE EDUCATION

 

Expt

Comp

p

Quality of course instruction
    A
    B
    C
    D
    F

(N=66)
23%
56%
18%
3%
0%

(N=82)
2%
40%
40%
10%
7%

 
 

<.001*

Student-friendliness of environment
    A
    B
    C
    D
    F

(N=66)
 
24%
48%
18%
9%
0%

(N=82)
 
2%
39%
32%
16%
11%

 
 
 

<.001*

Peer support
    A
    B
    C
    D
    F

(N=66)
67%
27%
5%
2%
0%

(N=82)
28%
52%
10%
9%
1%

 
 

<.001*

Investment in CHE education
    Very worthwhile
    Worthwhile
    Not very worthwhile
    Wasted

(N=66)
55%
42%
2%
2%

(N=82)
34%
57%
5%
4%

 

.06*


V. Discussion

One goal of engineering education is to equip students with knowledge of basic engineering content and ability to solve engineering problems. Another is to help the students develop and improve interpersonal skills generally considered essential for success in the workplace, including communication and teamwork skills.

To evaluate the longitudinal study, we must attempt to answer two questions:

  1. Did the experimental group learn more and achieve higher problem-solving and interpersonal skill levels than the comparison group?
  2. To what extent can observed between-group differences be attributed to specific features of the experimental course instruction?

The first question is difficult to answer definitively, for several reasons. No standardized chemical engineering tests exist that could have provided a basis for comparison of content learning. Moreover, since the experimental and comparison course offerings were offset in time by two years and taught by different instructors, there was no practical way to administer identical examinations to both cohorts (if for no other reason than the existence of exhaustive student archives of past examinations). Even if the latter measure had been possible it would not have been fair, since many of the skills taught and assessed in the experimental courses (e.g., interpreting familiar physical phenomena in terms of course principles, brainstorming, system troubleshooting, and problem formulation) were not part of the routine instruction in the comparison courses. We may, however, draw inferences about quality and extent of learning by examining patterns in between-course differences and comparing them with patterns observed in related published studies.

The question about attribution of performance and attitude differences to specific instructional features of the courses is even more difficult to answer, since the instructional approaches in the experimental and comparison course sequences differed in almost every aspect of presentation, testing, and grading. Moreover, the experimental group knew by their second semester in the study that they were being subjected to nontraditional instruction, so that the Hawthorne effect (wherein doing anything different from the norm may lead to positive responses) may very well have influenced the outcomes. Deciding how much each of these factors accounted for the observed performance and attitude differences between the two groups would be a hopeless task.

Fortunately, linking specific instructional methods to specific outcomes is not essential for the purposes of this study. Our goal was not to evaluate any of the individual components of the experimental instructional method, all of which have been validated in innumerable empirical studies and have solid foundations in educational and cognitive psychology. 17-22 Rather, the goal was to determine what would happen if the methods were used in combination over an extended period of time in an engineering curriculum, and this we are in a position to do.

In the remainder of this section we will suggest several inferences that may be drawn from the results summarized in the previous section and from the instructor's observations.

A. Retention in the chemical engineering curriculum was higher for the experimental group than for the comparison group.

This conclusion follows directly from Table 9. Relative to students in the experimental group, students in the comparison group were more likely to transfer out of chemical engineering, drop out of college, or still be enrolled in chemical engineering four years after enrolling in CHE 205. A much higher percentage of the comparison group switched or dropped out in good academic standing (16% of the comparison group vs. 5% of the experimental group), a result consistent with the relatively low ratings given by the comparison group students to their chemical engineering education (Table 11).

B. The experimental offering of the introductory course served as a better gateway to the chemical engineering curriculum than did the comparison offering of the course. The experimental course was a greater obstacle to students who lacked the motivation or aptitude to succeed in the field and provided a better preparation for the rest of the curriculum to those who passed.

Table 4 shows that 56% of the experimental group and 35% of the comparison group earned A's and B's in CHE 205, 12% of the experimental group and 46% of the comparison group earned C's, and 32% of the experimental group and 20% of the comparison group "failed." (Failure in CHE 205 is defined for the purposes of this study as earning a D or F or dropping the course, any of which would require repeating the course in order to remain in chemical engineering.)

Most of the experimental group students who failed CHE 205 gave up early in the course and most who completed the course requirements earned A's and B's. The large number of high grades cannot be explained on the basis of low standards: the concurrently high failure rate negates this hypothesis, and most students in the experimental group characterized the course as one of the most demanding they had ever taken.

Relative to their counterparts in the comparison group, experimental group students who passed CHE 205 earned higher grades in subsequent chemical engineering courses (Tables 4 and 5) and were significantly more likely to go on to graduate in chemical engineering (Table 9), despite their greater failure rate in the first course. The experimental group students also gave much higher ratings to the preparation for junior courses they received in CHE 205 and the course that followed it (Table 6). In short, most students in the experimental offering of CHE 205 either gave up early and failed or persisted and achieved a higher level of mastery of course material than most comparison group students did.

The superior academic performance of the experimental group is consistent with many other research studies of cooperative learning. In one study, significantly higher grades were earned in cooperatively-taught offerings of the introductory chemical engineering course than in a traditionally-taught offering with the same examinations.25 Johnson et al.18 report many other instances of students in cooperatively-taught classes outperforming students in traditionally-taught control groups. Given the immense body of research indicating that active and cooperative instructional methods improve learning, it would have been surprising if performance differences of the types observed in the present study had not appeared.

C. The experimental group developed higher critical skill levels.

Successful engineers must solve open-ended multidisciplinary problems, diagnose discrepancies between design predictions and actual process performance, and exercise the range of leadership, communication, conflict resolution, and other interpersonal skills required to work effectively on a team. Many published studies have demonstrated that cooperative learning facilitates development of most cognitive and interpersonal skills, as long as the skills are required to perform well in assignments and on tests.18 We would therefore expect greater improvements in the skill levels of the experimental group than in those of the comparison group.

Although we have no quantitative assessment data to confirm this expectation, we have several reasons to believe that it was realized. Skills inevitably improve through training, practice, and feedback. The experimental course instruction provided some explicit training and a great deal of practice and feedback in the cognitive and interpersonal skills in question and the comparison course instruction did not. The experimental course instructor observed dramatic improvements in open-ended problem solving, problem formulation, and teamwork over the course of the study, and the experimental group students gave their improvement in creative problem-solving ability significantly higher ratings than the comparison group gave theirs (Table 10). The superior performance of the experimental group in courses not taught by Dr. Felder (Table 5) suggests that once acquired, the improved problem-solving skills were transferable to a traditional learning environment.

It is interesting and initially surprising that the two groups gave themselves almost identical ratings on ability to work in groups, despite the extensive use of cooperative learning in the experimental course sequence. A possible explanation is that structured teamwork requires adapting to widely different ability levels, time schedules, and senses of responsibility among team members and constantly confronting and resolving differences of opinion. Learning how to do all that effectively is no less challenging than learning how to write and solve material balance equations, and the learning process is often difficult and frustrating. The experimental group knew all about the difficulties and many knew that they still had a great deal to learn about overcoming them. On the other hand, to many or most of the comparison group, "group work" simply meant ad hoc study sessions with friends. Since the students working in this manner never really understood how much was involved in effective team functioning, there was no reason for them to downrate their teamwork skills relative to the experimental group.

It was not until after graduation that many of the experimental group students understood how much they had learned about teamwork. In post-graduation surveys, most of them cite group work as the most valuable feature of their undergraduate education. Several have remarked they had to confront the same interpersonal problems on their jobs that they had confronted in the experimental classes, but unlike their counterparts from other institutions, they came armed with strategies for overcoming the difficulties.

D. The comparison group may have improved more in their abilities to solve computer problems and to work independently

Although the students in the experimental courses were constantly urged to outline all problem solutions individually before completing the detailed solutions in their teams, many did not regularly do so. Similarly, the tendency of many teams working on computer problems was to let the team members most comfortable with computers do most of the work. The comparison group's higher ratings of improvements in ability to work independently and to solve computer problems (Table 10) may therefore be justified. We believe that group homework might have been overemphasized in the experimental courses, and that a better balance between individual and group assignments might have made these courses more effective.

E. The experimental instructional approach led to better peer interactions.

In What Matters in College,7 Alexander Astin offers a wealth of evidence to support his assertion that "...the student's peer group is the single most potent source of influence on growth and development during the undergraduate years." He shows that frequency of education-related student-student interactions (including discussing course content with other students, working on group projects, and tutoring other students) correlates with improvement in GPA, graduating with honors, analytical and problem-solving skills, leadership ability, public speaking skills, interpersonal skills, preparation for graduate and professional school, and general knowledge.

The sense of community that developed within the experimental group was unlike anything the instructor had ever observed. Beginning in the first course, the students studied together, partied together, and when they were unhappy about something (like having to solve challenge problems to earn an A in a course), complained with remarkable unanimity. When asked about this unusual level of class coherence, many cited the group work that was a constant feature of the experimental courses and some remarked that they didn't know how they would have survived without the help of their teammates. In retrospectively evaluating their chemical engineering education, 67% of the experimental group and only 28% of the comparison group gave peer support the top rating of the alternatives provided (Table 11).

F. The testing and grading policies and procedures in the experimental course sequence contributed to the improved performance and attitudes of the students in the experimental group.

In the experimental courses, great care was exerted (not always successfully) to make sure that examinations were both rigorous and fair and that speed was minimized as a factor in test performance. The tests routinely included questions that required conceptual understanding and not just formula substitution, but students had extensive prior practice in such problems in class exercises and homework assignments. The instructor always made sure that he could complete the test in less than one-third of a normal class period, and when possible found a longer time period in which to give the test.

Although the tests in the experimental course offerings had (in the instructor's opinion) more high level content than traditional tests in the same courses, there were very few complaints about them. When asked midway through the third course of the sequence to cite the most likely reason if their performance failed to meet their expectations of themselves, 29% of the comparison group students cited unfair tests and only 2% of the experimental group did so. In response to the same question in the fifth course, the percentages citing unfair tests as the most probable cause of poor performance were 16% (comparison group) and again 2% (experimental group). As one of the seniors in the comparison group said in an evaluation, "I have yet to figure out why in every class since 205 and 225 professors give tests where the class average is 50. It doesn't make sense to me to fail 2/3 of the class on the tests only to curve the final grades, and it certainly did nothing for my self-confidence."

An absolute (criterion-referenced) system was used to assign letter grades in the experimental courses, so that in principle every student could have earned an A if they met the criteria for this grade. This outcome would be impossible if grades were assigned on a curve, as would the bimodal distribution observed in the experimental offering of CHE 205 (Table 4). Astin7-9 believes that the prevalence of curving in engineering relative to other fields helps explain why so many traditionally-taught students leave engineering for other fields. He notes that most engineering students were top scholars in high school and so are bound to be disappointed when evaluated by a system that requires most of them to get less than top grades. Their frustration in this situation could account for much of their dissatisfaction with their instruction and for the inclination many of them have to leave engineering, even if their academic performance has been adequate or more than adequate.

G. The experimental group and the instructor of the experimental courses each had superior educational experiences: A personal observation (RMF).

The individual strategies that constituted the experimental instructional approach-systematic use of instructional objectives, active and cooperative learning, criterion-referenced grading, and other methods described elsewhere4--were selected based on their proven effectiveness. References on educational psychology,21,22 success in college,7-9 and teaching effectiveness16-20,23,24,26,27 offer an abundance of evidence that the methods enhance motivation to learn, curricular retention, breadth and depth of learning, development of cognitive and interpersonal skills, self-confidence, and attitudes toward the subjects being taught.

As the instructor of the experimental courses, I infer from the quantitative results summarized above that the experimental group indeed received most of these advertised benefits, and it is those results that my colleagues and I offer as proof of the effectiveness of the experimental approach. My purpose in this section is to present qualitative observations that would not by themselves qualify as proof but are consistent with and therefore supportive of the quantitative results.

VI. Conclusions

The experimental instructional approach that constituted the basis of this study touched on almost every aspect of pedagogy. To summarize its features one final time, it involved

By design, the approach did not incorporate features that are costly or require unusual classroom resources like computers or tables designed for work groups. Any instructor at any institution should therefore be able to implement the approach.

The methods that constituted the experimental instructional approach have been shown in study after study to have positive effects on students' academic performance, motivation to learn, and attitudes toward their education and toward themselves. There was thus every reason to expect the experimental group to compare favorably with the comparison group, and indeed they did. Experimental group students generally earned higher course grades than comparison group students. Comparison group students were roughly twice as likely to leave chemical engineering for any reason prior to graduation and almost three times as likely to drop out of college altogether. Strong anecdotal evidence suggests that the experimental group outperformed the comparison group in developing skills in higher-level thinking, communication, and teamwork.

The attitudes of the two groups of students toward their education were dramatically different in many respects. Students in the experimental group gave significantly higher ratings to the quality of their course instruction, the student-friendliness of their academic environment, the level of peer support they enjoyed, and the quality of their investment in their chemical engineering education, and they were more than twice as likely to express an intention of pursuing graduate study in chemical engineering. The experimental group seniors felt that they had experienced much greater improvements in their basic and creative problem-solving abilities, while the comparison group seniors reported feeling greater improvement in their ability to solve computer problems and to work independently. (The latter two measures were the only ones on which the outcomes favored the comparison group.)

We set out in this study to demonstrate the positive effects on learning that might result from the repeated systematic use of well-established but nontraditional teaching methods with a cohort of engineering students, and we believe we have done so. There are limits, however, on the extent to which our results can be interpreted and generalized. All of the features of the experimental instructional approach undoubtedly contributed to the observed student outcomes, but there is no way to determine the individual contribution of each feature to each outcome or the synergy that may have existed among the features. Moreover, no educational environment can ever be precisely replicated: the same techniques used by a different instructor or by the same instructor with different students will inevitably produce different results.

We are therefore not in a position to predict that instructors who adopt the experimental instructional approach will see the same or even quantitatively similar results to those observed in this study. We can state with considerable confidence, however, that instructors who implement the defining features of the approach (preferably making the changes gradually rather than trying them all at once) will eventually see improvements in their students' performance. The more features they adopt, the greater the improvements they can expect. As they gain practice and learn more about the new methods, the skill levels of their students will continue to increase. The process is limited only by the ultimate potential of instructors and students to learn and grow-which is to say, it is limitless.

Acknowledgments

This work was supported by National Science Foundation Undergraduate Curriculum Development Program Grants USE-9150407-01 and DUE-9354379, and by grants from the SUCCEED Coalition and the Hoechst Celanese Corporation. We acknowledge with gratitude the significant and diverse contributions to the longitudinal study made by several colleagues. A conversation with Sue Kemnitzer of the National Science Foundation was instrumental in initiating the project and establishing its direction. Lynne Baker-Ward (Psychology, North Carolina State University) was an official coinvestigator, and Charles Hamrin (Chemical Engineering, University of Kentucky), and Phyllis Mohr (First-Year College, North Carolina State University) were unofficial coinvestigators. Meredith Mauney, Diane Hall, Cralen Davis, Hugh Fuller, and Krista Forrest provided invaluable assistance with statistical data analysis. Rebecca Brent, Ann Brown, Ruth Green, Sarah Rajala, William Rasdorf, Jonathan Sher, Norman Sprinthall, Gary Weinberg, and Ronald Wimberley reviewed manuscript drafts and made suggestions that resulted in significantly improved final versions.

References

  1. Felder, R.M., K.D. Forrest, L. Baker-Ward, E.J. Dietz, and P.H. Mohr, "A Longitudinal Study of Engineering Student Performance and Retention. I. Success and Failure in the Introductory Course," Journal of Engineering Education, vol. 82, no. 1, 1993, pp. 15-21.
  2. Felder, R.M., P.H. Mohr, E.J. Dietz, and L. Baker-Ward, "A Longitudinal Study of Engineering Student Performance and Retention. II. Differences Between Students from Rural and Urban Backgrounds," Journal of Engineering Education, vol. 83, no. 3, 1994, pp. 209-217.
  3. Felder, R.M., G.N. Felder, M. Mauney, C.E. Hamrin, Jr., and E.J. Dietz, "A Longitudinal Study of Engineering Student Performance and Retention. III. Gender Differences in Student Performance and Attitudes," Journal of Engineering Education, vol. 84, no. 2, 1995, pp. 151-174.
  4. Felder, R.M., "A Longitudinal Study of Engineering Student Performance and Retention. IV. Instructional Methods and Student Responses to Them," Journal of Engineering Education, vol. 84, no. 4, 1995, pp. 361-367.
  5. Board of Engineering Education-National Research Council, "Improving Retention in Undergraduate Engineering Education," Issues in Engineering Education: A Bulletin Addressing Culture Change in Engineering Education, vol. 1, no. 1, 1992.
  6. Heckel, R.W., "Engineering Freshman Enrollments: Critical and Non-critical Factors," Journal of Engineering Education, vol. 85, no. 1, 1996, pp. 15-21.
  7. Astin, A.W., What Matters in College: Four Critical Years Revisited, San Francisco, Jossey-Bass, 1993.
  8. Astin, A.W., "Engineering Outcomes," ASEE Prism, September 1993, pp. 27-30.
  9. Astin, A.W., "The Climate for Undergraduate Engineering Education: Results from a Recent National Study," Address presented at the Annual Engineering Dean's Institute, New Orleans, March 29, 1993.
  10. Moller-Wong, C. and A. Eide, "An Engineering Student Retention Study," Journal of Engineering Education, vol. 86, no. 1, 1997, pp. 7-15.
  11. Besterfield-Sacre, M., C.J. Atman, and L.J. Shuman, "Characteristics of Freshman Engineering Students: Models for Determining Student Attrition in Engineering," Journal of Engineering Education, vol. 86, no. 2, 1997, pp. 139-149.
  12. Hewitt, N.M. and E. Seymour, "A Long, Discouraging Climb," ASEE Prism, February 1991, pp. 24-28.
  13. Seymour, E. and N.M. Hewitt, Talking About Leaving-Factors Contributing to High Attrition Rates Among Science, Mathematics and Engineering Undergraduate Majors, Final Report to the Alfred P. Sloan Foundation on an Ethnographic Inquiry at Seven Institutions, Bureau of Sociological Research, University of Colorado: Boulder, April 1994.
  14. Tobias, S., They're Not Dumb, They're Different: Stalking the Second Tier, Research Corporation, Tucson, AZ, 1990.
  15. Cross, K.P., "On College Teaching," Journal of Engineering Education, vol. 82, no. 1, 1993, pp. 9-14 .
  16. Felder, R.M., "Reaching the Second Tier: Learning and Teaching Styles in College Science Education," Journal of College Science Teaching, vol. 23, no. 5, 1993, pp. 286-290.
  17. McKeachie, W.J., Teaching Tips: A Guidebook for the Beginning College Teacher, 9th Edn., D.C. Heath, Lexington, MA, 1994.
  18. Johnson, D.W., R.T. Johnson, and K.A. Smith, Cooperative Learning: Increasing College Faculty Instructional Productivity, ASHE-ERIC Higher Education Report No. 4, George Washington University, Washington, D.C., 1991.
  19. Bonwell, C.C. and J.A. Eison, Active Learning: Creating Excitement in the Classroom, ASHE-ERIC Higher Education Report No. 1, George Washington University, Washington, D.C., 1991.
  20. Meyers, C. and T.B. Jones, Promoting Active Learning: Strategies for the College Classroom, Jossey-Bass, San Francisco, 1993.
  21. Sprinthall, N.A. and R.C. Sprinthall, Educational Psychology: A Developmental Approach, 5th Edn., McGraw-Hill, New York, 1990.
  22. Haile, J.M., "Toward Technical Understanding." Part 1, "Brain Structure and Function," Chem. Engr. Education, vol. 31, no. 3, 1997, pp. 152-157; Part 2, "Elementary Levels," Chem. Engr. Education, vol. 31, no. 4, 1997, pp. 214-219; Part 3, "Advanced Levels," Chem. Engr. Education, vol. 32, no. 1, 1998, pp. 30-39.
  23. Woods, D.R. Problem-Based Learning, D.R. Woods, Waterdown, Ontario, 1994.
  24. Fogler, H.S. and S.E. LeBlanc, Strategies for Creative Problem Solving, Prentice-Hall, Englewood Cliffs, 1995.
  25. Morell de Ramirez, L. and C. Velazquez, "Enhancing Student Success in an Introductory Chemical Engineering Course: Impact of the Cooperative Learning Strategy," Proceedings, 1996 Annual ASEE Conference, ASEE, 1996.
  26. Eble, K.E., The Craft of Teaching, Jossey-Bass, San Francisco, 1988.
  27. Gronlund, N.E., How to Write and Use Instructional Objectives, 4th Edn., Macmillan, New York, 1991.