McMaster University

McMaster University

Development and Validation Project — Methods

This project included 3 phases: development, implementation and evaluation.

Ethics approval was received from both McMaster University and University of Ottawa prior to commencement of project activities.

Development

The development of the TOSCE stations was primarily informed by the experiences of the McMaster-Ottawa TOSCE project team (7 clinical multi-disciplinary investigators), which had approximately 2 years experience developing, delivering and evaluating TOSCE stations with interprofessional health science students. TOSCEs were developed using the following process:

Step 1: Topic Generation

The project team reviewed existing literature regarding clinical problems that were currently prevalent in primary care. Using these findings, as well as personal experience and knowledge of the clinicians on our team, a list of disease-specific medical topics relevant to primary care was developed.  Brief, 2-3 sentence, scenarios were scripted for each topic to clarify the extent of each clinical problem.

Step 2: Expert Consensus on Priority Topics

A two-step Delphi process (using a consensus driven format), was used to prioritize the topics with an invitation to suggest additional topics. A national interprofessional expert advisory panel were asked to rank the proposed topics in terms of their relevance to priority health problems in primary care and those that would be amenable to evaluation of team based competencies.  The panel was comprised of leaders within each profession typically present on primary care teams.  The 10 highest ranked topics were then selected for complete scenario development and evaluation.

Step 3: Station content and competency checklist

Station Content

The project team developed the 10 proposed topics into detailed scenarios which formed the basis for the TOSCE stations.  A variety of station formats were used and included simulated ‘live’ patient, video monologues, selective information sharing (via ‘vignettes of information), and discussions using case-based paper information (i.e. patient chart, hospital discharge summary).  Station content was meant to engage a variety of professionals and to elicit team collaboration and decision making.

Competency Checklist

There is currently no gold standard tool to assess attainment of interprofessional skills or team competencies.  Therefore, using a national set of interprofessional competencies (Curran et al; 2009), the team developed an observation checklist that included 6 core competencies.  These competencies adequately represented the objectives of the McMaster-Ottawa TOSCE tool and were reflective of current primary care educational and practice requirements. The focus of the McMaster-Ottawa TOSCE evaluation was on team collaboration rather than medical content or actions.


Implementation

Five community-based practices (three in Hamilton and two in Ottawa) participated in piloting the 10 McMaster-Ottawa TOSCEs and the observer checklist.  Each pilot team was asked to recruit four to six clinician participants from within their team.  A mix of disciplines was recommended.  The 10 TOSCEs were delivered to each team over two days (5 stations/day). For feasibility purposes, each station was assessed by two observers.  Using two observers per station also decreased the variation and increased consistency within stations.

The goal of these sessions was to pilot test the stations and the checklist, and was not to replicate how the TOSCE could be used by an actual team as a formative or even summative tool.  Each of the 10 McMaster-Ottawa TOSCEs was developed to be administered as a 20-minute station.  To evaluate whether station order affected team performance, the order of stations was randomly assigned by the project manager and the order differed across practices.  Because of the research design, the participating teams did not receive the usual recommended 10-minute formative feedback following each station. The overall study results, however, were shared.


Evaluation

The utility of the TOSCE was assessed across 5 dimensions:

  • Inter-rater and inter-station reliability was examined to determine whether or not the TOSCE provides the capacity to consistently differentiate between individuals that perform well within the team-based setting and those who perform less well. 
  • Construct Validity was assessed by examining the correlation between TOSCE scores and experience level (i.e. previous training in collaborative care).
  • Feasibility was assessed by monitoring the costs involved in mounting TOSCEs in these practice settings and through a post-project feedback survey of participants.
  • Acceptability of the process and ways in which it may be improved was considered through discussion with participants, conducted in the form of debriefing sessions and written feedback.
  • Finally, we surveyed participant’s post-TOSCE to determine whether or not the experience had changed the way they conceive of their practice or changed their day-to-day interactions with members of their team.

Previous: Development Introduction

Next: Development Findings

Valid XHTML 1.0 Transitional Level Double-A conformance, W3C WAI Web Content Accessibility Guidelines 2.0