JRSM >
Volume 102, Number 7 >
Pp. 287-293
1 Nuffield Department of Surgery, University of Oxford and Department of Neurological Surgery The West Wing, John Radcliffe Hospital, Oxford, UK
2 Department of Orthopaedic Surgery, Nuffield Orthopaedic Centre Windmill Road, Headington, Oxford OX3 7LD, UK
Correspondence to: Benjamin JF Dean bendean1979{at}gmail.com
 | SUMMARY |
|---|
 TOP SUMMARY IntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
Design and setting A total of 539 users across all surgical specialties (including 122 surgeons acting as assessors) were surveyed in late 2008 by online questionnaire regarding their experiences with the ISCP.
Results Sixty-seven percent had used the tool for at least one year. It was rated above average by only 6% for its registration process and only 11% for recording meetings and objectives. Forty-nine percent described its online assessments as poor or very poor, only 9% considering them good or very good. Seventy-nine percent rated the website's user friendliness as average or worse, as did 72% its peer-assessment tool and 61% its logbook of procedures. Seventy-six percent of respondents had carried out paper assessments due to difficulties using the website. Six percent stated that the ISCP had impacted negatively on their training opportunities, 41% reporting a negative impact overall upon their training; only 6% reported a positive impact. Ninety-four percent did not consider the trainee fee good value, only 2% believing it should be paid by the trainee.
Conclusions The performance of the ISCP leaves large numbers of British surgeons unsatisfied. Its assessments lack appropriate evidence of validity and its introduction has been problematic. With reducing training hours, the increased online bureaucratic burden exacerbates low morale of trainees and trainers, adversely impacting potentially upon both competency and productivity.
| Introduction |
|---|
|
TOPSUMMARYIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
Prior to MMC introduction of Specialty Registrar training in August 2007, British surgical trainees undertook basic and higher surgical training whereby progress was assessed using surgical logbooks of procedures, consultant reports and annual Records of In-Training Assessment (RITA). Proposals for competence-based training in surgery were conceived together by the Department of Health MMC Group, four Surgical Royal Colleges and then recently formed Postgraduate Medical Education and Training Board (PMETB) predicated upon tenets of trainee centred, service based, quality assured, flexible, coached, structured and streamlined training. The progression of a doctor through surgical training seems now largely based upon gaining satisfactory competencies in various workplace-based assessments, including ‘mini’ clinical evaluation exercises (mini-CEX), case-based discussions (CbD), direct observation of procedural skills (DOPS), procedure-based assessments (PBAs) and 360° appraisals using ‘mini’ peer-assessment tools (mini-PAT). Such assessments are administered using the ISCP which also seeks to hold an online portfolio including learning and educational agreements and surgical curricula relevant to specialty and stage of training.
To evaluate the impact of the ICSP on its users – surgical trainees, trainers and assessors – and their opinions of it, an independent survey of surgical trainees was undertaken.
| Methods |
|---|
|
TOPSUMMARYIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
|
| Results |
|---|
|
TOPSUMMARYIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
|
Figure 2 shows respondents' ratings of performance of the ISCP (a) registration procedure, (b) induction process, (c) online assessments, (d) mini peer-assessment tool (mini-PAT), (e) recording meetings and objectives, (f) logbook for procedures, (g) helpdesk and (h) overall user friendliness. The majority of respondents rated most aspects poorly, in particular the induction procedure, online assessments, mini-PAT and overall user friendliness. A total of 342 responders (75.6%) had carried out paper assessments as a result of practical difficulty associated with the online process.
|
|
|
| Discussion |
|---|
|
TOPSUMMARYIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
Hopefully the versatility and usability of the ISCP will continue to improve and the constructive criticism of trainees and trainers alike will be heeded by its developers, in particular now that its trainees are paying for it without prior consultation. The sustainability of the ISCP seems predicated upon securing long-term funding primarily for the trainee's salary, but also for trainers' commitments and associated educational resources. If no national funding transpires, many surgeons prefer such fees to be subsidized locally by deaneries. It is important that trainers are properly recognized and rewarded for the time that they spend assessing and supervising trainees if obliged to use increasingly time-consuming methods.5 A recent evaluation by the Association of Surgeons in Training (ASiT) estimated conservatively the upward spiraling costs of surgical training to the trainee to be £130,000 even before the introduction of MMC and the ISCP,6 ASiT's position on the ISCP is clear and stated below.7
‘Although a reasonable number of trainers have now signed up, there is a real feeling from the SACs and trainees alike that satisfaction is low, and more worryingly, that individual trainers are not sufficiently well informed concerning the completion of online assessments and as a result there is significant variability in the quality and validity of the information being obtained. We feel that until such a time as the system has a proven utility its development costs should not be passed on to trainees.’
It is possible that those most likely to complete the survey were those most dissatisfied with the ISCP. However, the findings above are unlikely to result from intransigence or resentment of the ISCP's mandatory introduction and fees, but moreover to reflect the genuine concerns of busy, motivated professionals seeking self-improvement, questioning both the validity and utility of the educational tools it contains and whether their use is an educational and efficient use of limited time. This year a consensus statement from the Association of Surgeons of Great Britain and Ireland (ASGBI) stated the following with regard to the soon to come into force European working time directive (EWTD).8
‘EWTD will have an immediate effect on both the breadth and depth of surgical training. The reduction to a 48-hour working week will reduce current elective training opportunities by 25% for trainees working a full-shift system with night cover. Six years of training on current 56-hour full-shift systems, will be equivalent to 7.5 years on 48-hour full-shift systems.’
One of the biggest challenges facing surgical training is how to cope with the massive reduction in training hours due to the EWTD. ASGBI also stated that ‘the completion of surgical training must be competency based and not time based’.8 Attempts to separate training time from competency like the ISCP might give the appearance of solving such problems. However in practice the two factors are inextricably entwined. Even with a more functional bureaucracy in place for trainees, increasing assessments and curricula may not improve training if training hours continue to fall.
Competency-based training is not without its problems. If applied inappropriately, it can result in demotivation, a focus on minimum acceptable standards, an increased administrative burden and a reduction in educational content.9 The ISCP relies on online assessments like the mini-CEX to assess the competency of trainees. The mini-CEX has been shown to be a reliable tool in strict exam conditions,10 far removed from its application in the ISCP, and in UK workplace-based assessments in general. Many surgical trainees' experience is that most assessments are filled out in batches in one sitting by trainers as an overall impression of the trainee rather than relating to individual cases (as evidenced by free text comments in our questionnaire). In such situations, the assessments represent little more than a trainer's report and not the rigorous quantitative assessment they purport to be. The evidence for workplace competency-based assessments is poor.11 Yet if they were to be removed from the ISCP they would leave a logbook of procedures not as comprehensive or as easy to use as its competitors and an inflexible record of meetings – a time-consuming online version of the old time-based system in other words.
The current shift in focus from experience towards so called ‘competency’ in UK surgical training threatens to add not in terms of quality, but by a profound negative impact due to its increased administrative burden.12 The likely barrier to progression from ST2 core surgical training to ST3 intermediate surgical training will remain failure to obtain the diploma of membership of a Royal College of Surgeons (MRCS). Recent changes to the MRCS have reduced the generality of the content, seemingly to cope with the reduced experience of candidates as training time becomes less and training more subspecialized from the outset. The regulation of the training content of so-called ‘training’ jobs is a key part of maintaining high standards; in recent years many core surgical training posts have metamorphosed from non-training senior house officer or clinical fellow jobs despite the training content of such jobs remaining unproven. Strict regulation of the training content of surgical training posts is imperative to ensuring that trainees are receiving the clinical and operative exposure they need. If not, no considerable volume of online assessments can repair the insult.
The survey highlights that there is widespread dissatisfaction among surgical trainees and trainers with both the ISCP and the trainee fee. Much needs to be done to improve the ease with which trainers and trainees can engage with its training administration. Nevertheless, improving the interface is unlikely to remedy the situation. Current attempts to rely upon competency-based assessments and training techniques must be seriously questioned. Junior trainees in largely service rather than training driven jobs now forced to foot the bill for a dysfunctional new bureaucracy find their low morale further deflated. De Cossart captures pithily the prevailing sentiment of those surveyed.11
‘Surgeons are chosen from among the highest achievers in school and university. They are individuals with a toughness of personality, fitness of body and an intelligent brain willing to learn and develop. They have been chosen from a highly competitive field. They must be given the best educational environment to allow them to develop into wise and trustworthy practitioners. Patients rely on this happening and rather assume that this is the case. However, those who are directing new educational programmes in surgery have become uncritically swayed by the superficially seductive approach of competency-based training.’
It remains wise to involve frontline surgical trainers and trainees when making sweeping changes to their training system. To quote Brown at the height of the now discredited UK Medical Training Application System (MTAS) debacle, ‘We are not against change or modernisation. We are passionate about quality, rigour, and humanity.’13 As MTAS revealed, the more detached and dictatorial any centralized administration governing passionate professionals becomes, the more likely its emperors are to keep changing their clothes.14 EWTD is among the factors auguring for changes in British surgical training, but the evidence is that many British surgeons are at best underwhelmed by the ISCP's alleged benefits and overwhelmed by its latest practical and financial burdens. Few surgeons would disagree with Tooke's plea, following his diligent collation of available evidence on MTAS and MMC, that ‘in medicine it is sometimes necessary to take decisions in the absence of optimal evidence in the interest of high quality outcomes’.15 Yet after initial decisive introduction of the ISCP, a referendum on its use and funding with adequate representation from frontline trainees and trainers appears both timely and appropriate.
Alongside the ISCP now undertaking a user forum questionnaire in May 2009, the results of which we hope to see painted warts and all, the debate has been advanced by ASiT's April 2009 response to the Eraut report, reiterating that the ISCP appears at present unfit for purpose.16 They provide constructive suggestions to improve many issues discussed here, most of which regrettably require those two commodities most precious to surgeon and government, respectively – time and money. Nonetheless, we remain hopeful of positive action being taken to restore, if not enhance, the quality of British surgical training.
| Footnotes |
|---|
DECLARATIONS
Competing interests EACP is an ST2 specialist registrar in neurosurgery and Member of the Royal College of Surgeons of England. BJFD is an ST2 specialist registrar in orthopaedics themed core surgical training and Member of the Royal College of Surgeons of Edinburgh. Both have used the Intercollegiate Surgical Curriculum Programme as trainees since August 2007 and pay the Joint Committee on Surgical Training's training fees. BJFD is a committee member of RemedyUKFunding Both authors are full-time salaried clinicians in the UK National HealthService employed by the Department of Health via local Trusts and Oxford Deanery. They were able to advertise their questionnaire to its target audience via RemedyUK and the Association for Surgeons in Training, but received no funding from external sources
Ethical approval All participants agreeing to the study gave ethical approval to its presentation and publication. No patients were involved in the study
Guarantor BJFD
Contributorship EACP analysed the results, produced the table and figures, and wrote the paper. BJFD designed and implemented the study and edited the paper. Both authors contributed equally
| Acknowledgements |
|---|
|
TOPSUMMARYIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
| References |
|---|
|
TOPSUMMARYIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
This article has been cited by other articles:
|
|
 |
|
  C. Munsch British surgeons' experiences of mandatory online workplace-based assessment - reply to authors J R Soc Med, September 1, 2009; 102(9): 357 - 358. [Full Text] [PDF]
|
|
|
|
 |
|
 S.-A. S Price Surgery's flawed e-portfolio BMJ, August 4, 2009; 339(aug04_3): b3142 - b3142. [Full Text]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
