JRSM >
Volume 101, Number 5 >
Pp. 252-258
The Heart Hospital Westmoreland Street, London W1 8PG, UK
Correspondence to: Dr Westwood markwestwood1{at}doctors.org.uk
 | SUMMARY |
|---|
 TOP SUMMARY IntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
Design Audit study of short-listed candidates for postgraduate specialty training programmes in the London Deanery.
Setting Postgraduate applications for the London Deanery.
Main outcome measures Satisfaction or otherwise with the application and selection process for postgraduate specialty training programmes amongst short-listed candidates in the London Deanery. Questions were asked under five categories: the applicant, the advertisement, the application form, the short-listing process, and the interview.
Results 89 of 118 forms were completed and analysed. Candidates thought the advertisement was clear on who to contact (97%), when short-listed candidates would be notified of their interview (66%) and when interviews would occur (93%). The design of the application form and the short-listing process both scored a median of 1 or 2 (strongly agree or agree) on all points. The interview process itself also scored well, with most candidates scoring broadly positively.
Conclusions As in the previous study, the overall response was broadly a positive one from the candidates' perspective, with the majority of candidates finding the system fair and objective.
| Introduction |
|---|
|
TOPSUMMARYIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
History
Interviews have always formed an important part of the process of selecting candidates at all grades within medicine, including selection into specialist training. The traditional approach has involved using a CV supplied by each candidate to short-list those for interview. Those short-listed would have an open interview, in which different questions were usually asked of different candidates. Potentially this leads to bias, with some candidates performing better than others based on the different questions they had been asked. There were also concerns that it may lead to canvassing, discrimination and patronage.4
The reforms of specialist training in medicine in the UK3 and the development of the specialist registrar grade (SpR) aimed to link the number of training posts to the predicted number of consultant opportunities. This gave increased importance to the selection process, since those selected could expect to become consultants after this period of training. The selection process for entry into this specialist training grade would need to be more rigorous and structured than previously.5 This new selection process would also need to be viewed as fair, open and accountable by both selectors and candidates. As a ‘high stakes’ point in the candidates’ careers, it needs to be fit for purpose and to accordingly select the candidates with the ‘best’ profile for the specialty.
There were several stages in the development of this new recruitment process. The first was to define the attributes required to be an NHS consultant, and then use this to develop a competency framework through discussion in a series of focus groups. This was then used to derive the person specification which clearly defined the various attributes to be assessed during the selection process.5 The point where each of the specific attributes of the person specification was to be assessed were clearly defined on the application form, at the structured interview, from the references, or a combination of all three. Although SpR interviews no longer exist, a similar process was used for Modernising Medical Careers (MMC), with explicit person specifications from which attributes to be assessed during the selection process are clearly defined.
The London Deanery procedure
As the structured application form was specifically designed to assess these various attributes, CVs were not accepted. By using a structured marking scheme, the application form was used to derive a short-list of candidates for interview.
The interview was structured with three panels of interviewers. At each panel, each candidate was asked the same set of initial stem questions. These stem questions were asked to each candidate by the same member of the interview panel, and preferably with the same wording. The stem questions were then followed up with more detailed ‘probe’ questions based on the initial responses.7
When the initial structured application process was devised and evaluated by the Institute for Employment Studies (IES), initial feedback suggested that both assessors and applicants found this new approach was fairer and more open as compared to the previous system.6 One of the recommendations of the IES, however, was that a periodic evaluation exercise would allow the system to evolve further. After the initial assessment from the candidates' perspective,6 there has been little further evaluation of this method of selection. We therefore devised a questionnaire to further evaluate the short-listed applicants' view of this process.
| Methods |
|---|
|
TOPSUMMARYIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
Several types of questions were used in the form. In the section ‘About you the candidate’, date of graduation and date of passing the MRCP examination were requested. The number of previous attempts at obtaining a SpR NTN and the number of previous interviews each candidate had attended were also requested. Candidates were also asked about other degrees they had obtained apart from the MBBS qualification (or equivalent). Candidates were asked about their current employment. Out of the four possible regions (North East, North West, South East and South West) each candidate was asked whether they would accept appointment to any of the four London areas. They were also asked if expressing this preference of region within the London rotation was the most acceptable and fair way to determine this. If they disagreed they were asked to give a preferred method. They were also asked specifically if returning to the previous system of interviews by the four separate London geographic areas would be preferable.
In the section entitled ‘About the advertisement’, candidates were asked if they were aware that an advertisement was due to be placed in advance, whether a clear point of contact was identified, if it was clear when a decision on short-listed candidates would be announced, and if it was clear when the interviews would occur.
For the above sections either the means and standard deviations or the percentage of candidates agreeing with the various statements was calculated, as appropriate.
For the other three sections – ‘About the application form itself’, ‘About the short-listing process’, and ‘About the interview’ – there were several statements which each candidate was asked to grade according to the following scale: 1, Strongly Agree; 2, Agree; 3, Neutral; 4, Disagree; 5, Strongly Disagree. As the responses to these questions appeared highly skewed, the median and interquartile range for each of these statements was calculated. At the end of each of these sections there was an open-ended question asking if there was any way in which the application form, short-listing process or interview could be improved. The individual responses have been described in the results section.
| Results |
|---|
|
TOPSUMMARYIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
About you the candidate
Short-listed candidates had graduated from medical school 7±2 years ago and had completed MRCP 3±2 years ago. The majority of short-listed candidates had obtained a BSc or equivalent ( Table 1). The higher degrees of the short-listed candidates are also summarized in Table 1, and the current jobs of the short-listed applicants are shown in Table 2.
|
|
Most short-listed candidates (77%) thought that expressing a preference using a form at the time of interview for one of the four London regions was acceptable. Three candidates suggested that a better method would be to indicate a preference well before the interview (such as on the structured application form) and three candidates thought that the preference should be indicated after the interview. Only two candidates suggested returning to the previous system of separate interviews for the four separate London Regions. Despite these preferences, the majority of short-listed candidates (81%) would have accepted an NTN from any of the four London geographical areas.
About the advertisement
53% of short-listed candidates were aware that an advert for London SpR applications was to be placed in advance. Nearly all short-listed candidates (97%) thought that the advertisement clearly identified a point of contact for prospective candidates. The majority of short-listed candidates (66%) thought that it was clear when the decision on short-listed candidates would be announced and almost all candidates (93%) thought that the advert clearly indicated when those interviews would occur.
About the application form itself
There was a favourable response to the design of the application form (Table 3). There was, however, some concern over the space available for completion of the application form, with 16% of short-listed candidates commenting that there was insufficient space to answer all the questions fully. Seven of these responses specifically commented about insufficient space for research publications. Two candidates thought that the application form should be a PDF folder.
|
|
|
| Discussion |
|---|
|
TOPSUMMARYIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
We have confirmed that short-listed candidates consider this structured process (both the structured application form and the structured questioning in the interview) a fairer and more objective assessment than the traditional CV and open interview. A similar result was obtained when this method of selection was introduced, with 76% of respondents thinking that the structured application form was fairer than short-listing by CV and 59% that the process better enabled candidates to demonstrate their strengths.6 All written documentation – both the initial advertisement and the structured application form – were easily understood.
There are a large proportion of short-listed candidates entering research prior to attempting to enter cardiology specialist training. There is clearly a need to try to further elucidate why candidates consider that it is desirable to undertake a period of research training in cardiology at such an early stage in their career, when many candidates will not have had experience of cardiology beyond general professional training. With the current application form there are several areas where points are scored either directly or indirectly for research. These include points for a higher research degree but also points for peer-reviewed papers and presentations at meetings. Several short-listed candidates obviously felt that they were more likely to be short-listed if they had done a period of postgraduate research; some candidates may have felt that postgraduate research is essential to gain the required number of points to ensure short-listing. The comments around insufficient space to list research publications would reflect this concern.
Postgraduate research experience is one of several desirable attributes of the person specification but is not an essential attribute. Credit should be given for previous high quality research. To maintain fairness to all candidates it must be possible to be short-listed without undertaking postgraduate research. When the structured application form was devised the space on the form for research publications was deliberately limited so that the emphasis would not only be on the number of publications per se but also the quality of those publications to maintain fairness and prevent undue weight being given to postgraduate research. Every comment made about insufficient space on the application form for publications was made by a short-listed candidate either in a postgraduate research post or who had completed a higher research degree. No short-listed candidate suggested a reason for increasing the space available for research publications or why this may be fairer to all candidates. One candidate suggested that the form should be more explicit in establishing the personal involvement of the candidate in obtaining their research funding. This may be an important point, as credit should be given to those whose research proposal has undergone peer-review and has attracted competitive funding only where they were actively involved in this process. This issue of research funding cannot have been a major concern, as overall candidates overall felt the application from to be fair and objective.
The majority of short-listed candidates had previously applied for cardiology specialist training and a significant proportion had previously been interviewed. The selection process was changed in 2004 with the introduction of Pan-Thames Interviews twice yearly replacing separate interviews in each of the four areas of London at various times of the year. Only two candidates suggested a return to this system of separate interviews for the four separate London areas.
The short listing process was felt to be fair and objective. A few comments were made about the time it took to notify successfully short-listed candidates or that there was no explicit date given when the results of short-listing would be available. Although it is preferable to notify candidates about the results of short-listing as soon as possible, it is important that all candidates are notified at the same time for the sake of fairness. Only one candidate suggested that the system was not as objective as it should be, but without any suggestions as to how the system might be made more objective. One candidate suggested that increased weighting should be given to the references. In order to maintain objectivity and fairness, references were excluded from the marking scheme when the structured application form was devised, as they are subjective opinion.
The interview process itself was also felt to be generally fair and objective. It is of interest that inclusion of presentations of any kind was not considered to be of benefit. We are unable to provide the reasons for this from our study. Several short-listed candidates suggested increasing the overall time of the interview either by increasing the number of interview panels or having a longer time with each interview panel, but overall the structured interview process was felt to be fair. With a longer time per panel there is the danger that more follow up ‘probing’ questions after the initial stem question will be asked. This could lead to substantially different questions being asked of each candidate and the structured interview has been devised in such a way to try to limit this to be fair to all candidates. The option of more interview panels to assess other qualities such as an aptitude test or some form of psychometric testing (as suggested by one candidate) could be explored in the future as long as they can be shown to be a valid, reproducible and fair method of assessment. Simulated clinical scenarios involving trained actors would be a further possible way of extending the scope of the structured interview process.
Limitations and suggested further work
Some caution must be used in assessing the objectivity of the short-listing process from the responses, since all respondents had succeeded in being short-listed. To gain a better overall view of this, the views of all those filling in an application form would be required, and could be addressed in the future. One further area that was not explored was the feedback that all candidates (successful or otherwise) receive after the results are announced to each candidate. We chose not to include this as it would mean that the questionnaire would be completed after the results of the interview were known, and this could introduce a large potential bias. This study was carried out before the introduction of MMC. With the significant changes to medical training that have occurred with MMC, further work is required to ensure that from the candidates' perspective the process continues to be fair and objective.
Summary points
In summary, this questionnaire has shown that from a candidates' perspective the current structured procedure does represent a more objective and fair assessment of candidates. It shows similar results to those found when the structured application form and interview were first introduced. Continued monitoring of this will be valuable in the future, especially with the proposed changes to specialty training through the MMC programme. This includes a major change to the way in which doctors will be recruited into specialty training, using an electronic application portal, national person specifications and application documentation for each specialty, as well as a more standard approach to selection interviews.
| Footnotes |
|---|
DECLARATIONS
Competing interests None declaredFunding None
Ethical approval Not applicable
Guarantor MAW
Contributorship All authors contributed equally
| Acknowledgements |
|---|
|
TOPSUMMARYIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
| References |
|---|
|
TOPSUMMARYIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
CiteULike 
Complore 
Connotea 
Del.icio.us 
Digg 
Reddit 
Technorati What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
