ARCP and Interim Reviews

The School of Surgery expects that a minimum number of the following assessment tools are carried out by the time of their interim review:

– Six CBD (three to be done with a consultant)

– Six Mini-CEX (three to be done with a consultant)

– Six DOPS (CT1) (three to be done with a consultant) or PBAs (CT2) (three to be done with a consultant)

By the time of ARCP, trainees must have completed a minimum number of the following assessment tools and provide evidence of:

– 12 CBD (six to be done with a consultant)

– 12 Mini-CEX  (six to be done with a consultant)

– 12 DOPS (CT1 six to be done with a consultant)

– 12 DOPS/PBAs (CT2 – six to be done with a consultant, PBAs only count with a consultant)

– One Mini-PAT (compulsory in CT1 and CT2)

– Six Teaching Days

– Leadership Topic

– Non-Technical Skills for Surgeons (NOTSS) Assessment

– Form R Parts A & B

The actual number is predicated on training delivering curriculum requirements, to enable the trainee to gain sufficient depth of experience. The trainee’s Assigned Educational Supervisor (AES) has a key role in judging whether the trainee requires more than the minimum number.

During training, WBA is part of a learning cycle, providing the basis for on-going dialogue and regular feedback. Following an assessment, the trainee should reflect on feedback and augment practice with further study in a safe environment e.g. skills lab or coaching. Emerging problems should be followed by targeted corrective action and interim assessments.

One point worth emphasising relates to assessments of technical procedures (i.e. Direct Observation of Procedural skills (DOPS) and Procedure-based Assessments (PBA). During training, a trainee will be learning a range of technical procedures simultaneously, and the indicative number of assessments is for each one of these technical procedures, so that progression can be gauged for all the procedures that the trainee is seeking to learn at that point in time.

A range of assessments indicates whether the trainee is making satisfactory progress in terms of meeting overall educational goals. When a trainee has demonstrated achievement of one level of curriculum competencies, it is expected that they will show they are maintaining that level of competence while being assessed on the next level of competencies.

Assessment for learning should be seen as an on-going process. When we discuss frequency below we refer to the minimum standards expected for trainers and trainees to achieve. The minimum standard is defined as that which is required to provide a reliable assessment. Ideally, assessment for learning should be occurring as frequently as learning opportunities arise. Every encounter between trainer and trainee should be seen as a debriefing and feedback opportunity i.e. regular feedback becomes the norm.

Spreading assessments, e.g. at least one per week with a mix of type throughout the placement, optimises the effectiveness of WBA and improves reliability. Trainees should request assessment for any new task that they undertake as soon as possible. On-going constructive feedback is conducive to reflection and helps keep progress on track; it also provides the AES with information about the trainee’s rate of learning and developmental needs.

Observing the trainee with several patients is desirable from an educational perspective because different patients in different settings require different skills from trainees and this significantly broadens the range and richness of feedback they receive. The only implication of an individual assessment that is unsatisfactory is that it should be repeated at a later date, ideally by another assessor.

Similarly, trainees should be evaluated by different members of their team who have their own expertise, strengths and perspectives. Feedback from several different assessors will benefit trainees as it improves reliability. Trainees should actively seek feedback from the whole team, including patients and their carers. The philosophy of what can be learned to become better should be at the core of professionalism.

The primary function of the rating scale is to inform the trainee and the trainer about what needs to be learned; therefore rating trainees accurately is critical to the success of WBA. It is essential that everyone who is responsible for assessing trainees is trained to ensure they understand how to use the rating scales so that ratings can be applied consistently.

The standard against which the trainee should be rated differs according to the assessment tool.

Most of the assessments compare the trainee’s performance against the curriculum standard of a doctor at the end of that particular stage of training, rather than at the level of the assessor’s personal expectation of the trainee at that point in time. Assuming that WBA appropriately challenges the trainee, the norm for a trainee at the start of a stage would be a rating of below expectations (rating 1-2). A rating of meets expectations (rating 4) should be achieved by the end of the training stage. Ratings of above expectations (rating 5-6) indicate that the trainee can manage all the competencies, including complications without any supervision. It would be rare for trainees to attain this level in the early years.

PBAs rate trainees’ performance against the standard required for CCT, both for the individual items and the global summary. A satisfactory standard for individual items in a PBA will often be achieved at an early stage but a global rating of level 4 will only be consistently achieved after a number of procedures have been undertaken. Repeated assessment should show evidence of progression towards this level. The number of procedures required to reliably achieve level 4 will vary according to the trainee and to the complexity of the procedure, so there is no current ‘indicative number’ of cases that are required for any procedure to demonstrate competence. Early feedback of areas requiring development are of value and a less than ’perfect’ PBA is to be expected when learning a new procedure.

Attainment of the top rating for a particular assessment does not mean that there is no need to undertake further assessments of that type. As learning and training tools, the assessments continue to have value, and particularly with the assessments of technical skills, it is important to confirm that competence has been maintained.

Verbal and written interpretation of the ratings enhances the validity of WBA and ensures that the trainee receives the type of constructive criticism that should result in a reduction of errors and an improvement in quality of care.

Feedback needs to be couched in a way that leads to the improvement of the trainee’s performance rather than damaging the trainee’s confidence. A useful starting point for discussion is to invite the trainee to provide their own insight into the events and their own performance. The feedback noted in the free text feedback box should include what went well, what areas need to be improved and an agreed strategy for meeting the trainee’s learning needs.

Key points

– Trainees and trainers need to be actively motivated to use WBA methods for learning and as an indicator of progression, rather than as ‘mini-exams’.

– Assessments should be spread throughout each placement with evaluations from different assessors with different patients in different settings.

– Assessments of new tasks should be undertaken as soon as possible to aid learning through constructive feedback.

– Assessments should be punctuated with reflection on feedback and further study.

– Assessments that are consistently below standard should be tackled with targeted training and additional assessments.

– Both trainers and trainees should be prepared to trigger and accept the triggering of WBAs.

– It is essential that everyone who is responsible for assessing trainees understands how to use the rating scales and how to provide constructive feedback aimed at helping trainees to learn.