EVALUATION IN A PROBLEM-BASED COURSE: CONTRASTING VIEWS OF STUDENTS AND TEACHER

K. Rawnsley, R. Spaziani and P.K. Rangachari
Honours Biology-Pharmacology Co-op Programme, McMaster University, Hamilton, Ont. Canada L8N 3Z5

Originally published in PROBE, the newsletter of the Australian Problem-Based Learning Network, No. 12, November 1994

Introduction | Evaluation of Tutorial Performance | Problem Write-Ups | Critical Analysis | Essay | Triple Jump Exercise | References

Introduction

Educational enterprises hope to produce changes in knowledge, skills or attitudes, preferably all three. These hopes are best realised if the recipients are actively involved in the process. Problem-based Learning is but one form of promoting student-centred learning (Boud,1988). Although, in principle, problem-based learning could be achieved in different ways, the small-group, tutorial based variety has become the most widely discussed format (Boud & Feletti, 1991; Branda, 1990). Despite the popularity of the approach, lingering concerns exist. Appropriate evaluation of PBL is a particularly vexing issue. In this brief essay, we seek to provide some information on that score.

A particular PBL course in pharmacology will be described. This course was developed by one of us (PKR) as part of a larger programme in Pharmacology (Rangachari, 1991; 1994). Each of the specific evaluation techniques used in that course will be described by PKR who will also attempt to define the underlying rationale from the teacher's perspective. Then KR and RS who as 3rd year undergraduate students took this course in the 93-94 cycle will describe their experiences; this will constitute the student perspective.

The Honours Biology-Pharmacology Co-op Programme:

The above programme was set up as a joint venture between the Faculty of Health Sciences and the Department of Biology. Students enter the programme after having completed at least two years of the regular biology programme where the courses are taught in the conventional lecture-based format. Once they enter the Honours Biology-Pharmacology Programme, they continue taking regular courses in biology, but all the pharmacology courses are run in the standard, small group tutorial format. The courses are carefully sequenced so that in the first year (i.e. 3rd undergraduate year), the students take an Introductory Pharmacology Course (hereafter referred to as 3A06) and a complementary laboratory course. In subsequent years, they take courses in Signal Transduction Mechanisms, Psychopharmacology, Toxicology, Drug Design and Pharmacoepidemiology. The academic terms alternate with work terms that are spent in Industry, Academia and Governmental Agencies. The combination of work terms and courses is designed to give students a broad-based education ranging from Molecular Pharmacology on the one hand to Social Pharmacology on the other.

The Introductory Pharmacology course that will be discussed has two major objectives (Rangachari, 1991). The Process objectives are stated as follows: Given a problem in Pharmacology, the students should be able to generate issues, organise relevant issues into learning tasks, seek appropriate information from a variety of sources, analyze critically the information obtained, synthesise the information into a coherent framework, share the information obtained, facilitate the learning of others and assess their own performance on an ongoing basis and help peers in assessing theirs. The Content Objectives include those that are conventionally dealt with in such courses and involve delving into pharmacodynamic and pharmacokinetic issues (See Rangachari 1991).

Evaluation: The Teacher's View

Multiple evaluation procedures were incorporated to give students an opportunity to demonstrate their strengths on at least one if not more evaluation tools. Since the programme stressed tutorial participation, it was important to emphasise that component and therefore 40% of the marks were allocated to tutorial performance. The other exercises were designed to explore more tightly the acquisition of a pharmacological knowledge base.Top of page

1) Evaluation of Tutorial Performance

The Teacher's View

Amongst the central tenets of this approach is the belief that process and content are inextricably linked, that what is learned stems closely from how it is learned. Promotion of active learning by this process requires that students take responsibility for their own learning and are able to seek, synthesise and integrate information as well as communicate it effectively to their fellow students. Formative evaluation becomes a crucial component of such a programme. In the conventional small group format, time is set apart for an evaluation of the tutorial process at the end of each tutorial. The amount of time spent is variable depending on the attitudes of the tutor and the students. Often this process is short-circuited since few written notes are kept. To formalise this process, a specific form was developed. This was initially developed with student input from a previous class. Specific categories were chosen for evaluation. These were Responsibility, Information, Communication, Critical Analysis and Self-Assessment Skills. Under each category a set of statements were given. The students indicated the strength of agreement with each statement on a 5 point scale ranging from 1 (strongly disagree) to 5 (strongly agree). Once they had completed the form, they were asked to evaluate their performance on that particular tutorial as being below average, average, above average or outstanding. These forms were kept as ongoing records of student evaluations and progress (Rangachari & Crankshaw, 1992; Bajcar, Winslade & Rangachari, 1994).

At the end of each term, a session was devoted to summative evaluation. Students were given beforehand a table with the names of the students and 4 categories (Responsibility / Information / Communication / Critical Analysis). They were asked to grade each other on the above categories, 5 marks being allocated to each. These sheets were brought to the discussion. At that session each student was evaluated in turn. The marks given were read out and tabulated openly on the board. The student had an opportunity to indicate whether he or she felt that the marks given were fair. The marks were averaged. In this section, the tutor's marks were averaged with those of the other students. It is my impression based on several classes that this system does work reasonably well. The students are able to discriminate amongst different categories. Thus there are students who receive fairly high scores on providing information but lower scores for critical analysis and vice versa. Others receive high scores for providing information and lower scores for clarity of communication.

The Student View

There are primarily three components to this evaluation; the written evaluation, the oral self evaluation and the group feedback.

The written evaluation allowed the tutorial's performance to be quantified such that an individual could assess their progress in definable terms. The oral evaluation allowed the students to voice their opinions of their performance, forcing the students to honestly assess their performance, making them accountable to their peers and making them defend and perhaps change their evaluation. This oral evaluation stimulated discussion of both group and individual performances such that immediate feedback was given to the individual concerned and also provided an opportunity for constructive criticism.

The summative evaluation at the end of the term was an excellent opportunity to identify trends in personal performances that may not have been evident on a day to day basis.

Group dynamics and personal interaction skills were adequately and thoroughly addressed by the formative evaluation. However, one aspect, content, in terms of the presentation of information, was not adequately addressed by the formative evaluation. Although the written evaluation attempted to distinguish between the amount of information a student provided and the quality of that information, this was not easy to distinguish in practice. Relatively quieter students who may have made briefer but significant comments were undervalued. This is an area where tutors need to be particularly vigilant to emphasise to the students the need to appreciate such contributions.Top of page

2) Problem Write-Ups

The Teacher's View

Students deal with nine separate problems in this course. At the end of each problem, students are expected to submit a brief write up. The purpose of this exercise is to provide students with an opportunity to summarize their learning and provide them with a useful log of the content component of the course. For the tutor, it provides a valuable insight into what the students have learned and helps him/her identify factual or conceptual errors.

The Student View

Writing the problem summaries provided an excellent opportunity to bring together all the information that was learned and see how all the components fit together. It was also an opportunity to ensure that all the concepts covered were truly understood. If something was still unclear, students could seek out more information in the literature or consult peers and tutors. A somewhat more subtle yet significant outcome of writing a problem summary was that it allowed students another chance to review the material before the next problem was attempted, aiding factual recall.

The summary, as a process, inherently allows the development of basic writing skills. In addition to evaluating the content of these summaries, emphasis was also placed on the ability to create concise yet coherent and thorough papers.Top of page

3) Critical Analysis of a Published Paper

The Teacher's View

PBL attempts to promote the development of the life-long learner, which demands that the student learns to critically assess information. Much of this information in an experimental subject such as pharmacology is published as research reports in peer-reviewed journals. A formal exercise was developed to train students in this matter. Each student was given two handouts. The first described the general format of a research publication with comments on each of the specific components (Introduction, Methods, Results, Discussion, References Abstract and Acknowledgements). The second was a specific checklist which enabled the student to ask particular questions of each component. Students were expected to use the checklist to analyze critically a brief report from a current journal in Pharmacology. Usually the article chosen used a technique that the students were familiar with from a companion laboratory course they were taking (Rangachari & Mierson, 1995).

The Student View

Many courses expect primary literature to be reviewed to obtain the most up to date information in specific topics. What most of these courses fail to do is point out the inconsistencies of these papers. Many students believe implicitly in the conclusions drawn from research papers. The critical analysis component of this course highlights these inconsistencies and provides students with a mechanism to truly assess the information provided and place it in an appropriate context.

The course accomplishes this by giving the students a particular research paper and a set of general guidelines or questions that the student can ask about the paper. This provided a structured approach to the process of critically analyzing a paper allowing for a general applicability.Top of page

4) Submission of An Essay on a Clinical Trial

The Teacher's View

Amongst the advantages of PBL courses is the opportunity that students have to pursue subjects that particularly interest them. These belong to the category of "expressive" objectives discussed by Eisner (1985). Although the course described dealt with the principles of pharmacology, it was necessary that students acquired some familiarity with applied aspects. Several problems in the course dealt with issues such as the passage of a drug from the bench to the market. The special essay permitted students to explore an area of pharmacology or therapeutics that was of special interest to them. They were instructed to submit a proposal for the clinical trial of a drug. They were given the liberty to invent their own drug or even their own disease. However, it was expected that the essay would provide evidence that the student was familiar with the general principles of the design of such trials. The assessment was based on a clear statement of objectives, logical development of the proposal, appropriate referencing and above all imagination.

The Student View

A particular failing of most science courses in general is their failure to permit creative writing. The emphasis on fact finding and precise representation of those facts does not allow for creative expression. This is unfortunate since creative thinking is an essential part of problem solving. The clinical trial essay component of the course managed to combine the fact finding of a clinical trial process in the development of a drug, with the option of creativity by applying the process to a fictional drug and or disease. The effect of this was three fold: relevance to potential work terms, promotion of writing skill and the development of creative thinking. Creativity is a skill which must be used, developed and promoted.Top of page

5) A Group Triple Jump Exercise

The Teacher's View

This was a formal tri-partite exercise that appeared to model the scientific process. In phase 1, students were presented with a specific written problem (clinical situation, experimental data) and given 20 minutes to provide one to three possible explanations for the observations made. In the second phase (20 minutes), they were asked to choose one of their explanations and design an experimental test or tests to verify their explanations. In the third phase (2 hours), they were provided ancillary information and asked to re-assess their explanations and experimental tests in light of the new information. Each of these three phases were evaluated separately. Thus it was quite possible for a student to frame a reasonable explanation but not be able to test his or her hypothesis adequately. On the contrary, a student might not have a particularly strong hypothesis but might be able to provide an excellent test of that hypothesis. The third phase provided the students an opportunity to salvage their performance if they had not done particularly well in the other two phases.

The Student View

The triple jump exercise was essentially an opportunity for the students to apply both the knowledge and the learning skills acquired from the problem-based tutorials.

The first phase served to test the student's ability to critically analyze a pharmacological problem. In order to formulate valid hypotheses, it was necessary to quickly appraise the information presented in order to identify the key features and salient trends. This stage relied heavily on the ability to apply fundamental pharmacological concepts to both understanding the scenario and dissecting out the relevant information.

Although some students have argued that this necessity to perform in such a limited time frame (20 minutes) is not representative of the everyday experiences of research pharmacologists, we would argue that it is in fact valuable. The ability to be observant and to quickly recognize key information is a vital skill to any science student who on a daily basis encounters enormous amounts of information, and is a skill which has proven very useful in our work term placements.

The second phase tested the ability of the student to think logically. Having selected a hypothesis, it was then necessary to devise a logical, step-wise experimental test that would categorically refute or support the postulated hypothesis. This stage places the onus on the student to be creative and to draw on all their previous knowledge of experimental concepts.

The third phase represented an exercise in critical analysis. Presented with the complete information on the topic, the student then had to identify what the 'actual' crucial elements in the problem were and evaluate how their assertions complemented this information. This phase represented a fantastic opportunity to obtain immediate feedback on the individual's triple jump performance. The student at this point can critically assess their own performance. It pinpoints which thought processes were valuable in assessing the problem and which required improvement. And once again, the student has the opportunity to practice his or her written communications skills in stating how and why their speculations were accurate or inaccurate.

Conclusion: The evaluation procedures used in a specific problem-based course have been discussed from the perspectives of the teacher and the student. It is interesting to note the concordance between the expectations of the teacher and the fulfilment of the student. The dissonances that appear are relatively minor and are amenable to correction. We firmly believe that the student is the raison d'Ítre of any educational endeavour, and thus any procedures that permit then to express their individuality and grant them greater autonomy are to be encouraged. In this context, the provision of multiple evaluation procedures appears fairer to the student, permitting them an opportunity to display their strengths. In the experience of the students, the course provided them with skills in information retrieval, communication and self-assessment that were of immense benefit to them in other courses. It is important to emphasise that ranking and grading students in the tutorial setting did not "destroy" the tutorial process. The arguments presented here may be useful for both students and teachers engaged in the supremely rewarding task of dispelling the penumbra and umbra of ignorance.Top of page

References (use Back button to return to your place in the article)

Bajcar, Winslade, Rangachari (1994) Proc. 7th Show & Tell Conference, U. Guelph, pp 9-10.
Boud (1988) in Developing Student Autonomy in Learning (2nd edn.), Boud, D., ed, pp 17-39, Kogan Page.
Boud, Feletti (1991) in The Challenge of Problem-based Learning (Boud, Feletti, eds) p 14, St. Martin's Press.
Branda (1990) J. Dent. Educ. 54, 1-2.
Eisner (1985) The Art of Educational Evaluation: A Personal View, The Falmer Press, London & Philadelphia.
Rangachari, (1991) Am. J. Physiol. 260, S14-21
Rangachari (1994) Trends in Pharmacol. Sci. 15, 211-214
Rangachari, Crankshaw (1992) Proc. 5th Show & Tell Conference, U. Guelph, pp 125-126.
Rangachari, Mierson (1995)  Am. J. Physiol. 268: S21-S25.


P.K. Rangachari
McMaster University