The World Health Organization (WHO) advocates that medical training should develop socially accountable health professionals with the required skills.^1,2 Socially accountable medical training requires education and service activities that address the priority health concerns of the communities being served.² Trainees’ ‘social responsiveness’, an essential element of social accountability,³ refers to their ability to integrate the complex skills required to attend to patient needs and improve patient outcomes in the community.

The recent shift to work-based learning (WBL) and workplace-based assessment (WPBA) aligns with the call for greater social accountability in medical training. Training in authentic clinical contexts promotes professionalism, communication, teamwork and interprofessional collaboration and enhances self-directed learning, thus preparing trainees to be responsive to communities’ needs.^4,5 Despite the benefits of WBL, medical schools may not be equipped to assess the complex integration of knowledge, skills and attitudes in authentic settings and have been criticised for not linking the phases of trainee learning with patient outcomes.⁵ The educational impact of WBL has been explored in high-income countries, but the use of WPBAs and the types of assessments used for and of learning have not been adequately researched in sub-Saharan Africa. This article describes the evidence of learning in WPBAs in a Family Medicine training programme in South Africa (SA).

Work-based learning shifts the focus from theoretical learning to learning in and from practice.⁶ Work-based learning also broadens the scope of educational assessment from assessment of learning (AoL), summative components with pass/fail decisions,⁵ to assessment for learning (AfL).⁷ Assessments for learning form part of day-to-day clinical practice: seeking, reflecting and responding to information from observation and dialogue to promote learning.⁸ Assessment for learning provides meaningful narrative feedback to enhance the learning experience.^7,9 Assessments of learning are high-stake assessments focusing on achieving learning outcomes at the end of a programme^7,10 for accountability, ranking and certification of trainee’s achievement of competence.¹¹

The distinction between AfL and AoL is complex, making it difficult for trainers to discern how to use the different assessment types effectively.¹¹ Workplace-based assessments captured in learning portfolios (LPs) provide adequate evidence of trainees’ competence and are collectively used for the summative AoL.⁷ Given that WBL and WPBA are recent developments, it may be more difficult for trainers to use AfL and AoL to the trainees’ best benefit.

In programmatic assessment, defined as ‘a consciously designed assessment system in which the longitudinal development is visible to the learner as usable feedback which provides rich data for informed, holistic decision on making learner progression’, individual data points from low-stake AfL accumulate into multiple data points for high-stake AoL.⁴ Learning portfolios, introduced in the United Kingdom in 1996 for postgraduate general practice vocational training,^12,13 are now widely used as a longitudinal tool in programmatic assessments.

Learning portfolios offer several benefits for trainees’ longitudinal development. Portfolios may include evidence of personal and professional WBL,¹⁴ assessments of soft skills, like professionalism, communication, teamwork and ‘hard’ clinical procedural skills.^4,15 Soft skills are better learned and assessed when opportunities are provided to address the complexities of clinical practice in the workplace.^4,14,15 Learning portfolios provide a systematic overview of the tasks, expected levels of competency for the required tasks, perceived levels of competence (ability to perform the tasks) and areas needing improvement.¹⁵ Competency cannot be measured, but competence, an attribute of the person performing the task, can be measured.¹⁶ Learning plans, an essential component of LPs, encourage trainees to track their educational progress and stimulate reflection and feedback as critical skills in self-directed learning.^13,17 Supervisors are expected to assist trainees in developing learning goals and support them in achieving their learning plans.^18,19 The most critical factor influencing LP use in high-income countries was supervision that stimulates trainees’ reflection and promotes deep learning.^20,21 The organisation of portfolios also affects their efficacy.²¹ Trainees require clear guidelines on the portfolio structure, format and content and how to avoid excessive paperwork.²¹

Workplace-based assessments are best used as part of a programmatic assessment approach; no single assessment method embodies all the required educational characteristics, such as reliability and validity, educational impact and acceptability.²² However, different WPBA tools have varying educational impacts across clinical contexts.^23,24,25 In high-income countries, the effectiveness of WPBA tools relied on prior training on using the tools and the contexts where assessments were conducted.^22,24 One factor affecting WPBA is the lack of consensus on the number of assessments used as AfL.²⁴ Other factors arise from the trainee-tool interactions. For example, trainee-assessor relationships could promote lenient marking.²⁴ Despite the challenges associated with WPBAs, the trainees and trainers agree that WPBAs provide valuable narrative feedback and augment trainee self-reflective skills, thus improving learning opportunities.²⁶

Potential barriers to effective LP use include the paperwork required, finding senior staff to supervise procedures, ineffective supervision, the lack of faculty engagement²¹ and ticking boxes to indicate whether skills were performed.^17,27,28 Ticking boxes does not provide feedback on the quality of performance.²⁰ Other barriers that have been identified were the need for multiple assessors, inadequate supervisor and trainee training on WPBA tools, low trainee confidence and whether WPBAs are conducted for learning or of learning.^13,21,24 In one study, trainees were concerned that delayed feedback after WPBAs would impact their results – supervisors could not always remember individual trainees.²⁸

While WBL theories and models have been developed, variations in how trainees and supervisors utilise WPBA tools across settings need further research.¹³ Workplace-based assessment tools have been extensively investigated in high-income settings like the Netherlands and the United Kingdom, but there is a paucity of studies exploring their effect on trainee performance in low-middle-income contexts.²⁹ There is a need for further characterisation of LP use, purpose, structure, evidence for mentoring and assessments across different contexts.²¹ Some authors have suggested a need to explore the influence of context on LP use to understand why implementation failed in some contexts and to identify and interrogate improvements.²¹

Learning portfolios were introduced SA as a WPBA tool in postgraduate family medicine (FM) training in 2012.³⁰ The Colleges of Medicine of South Africa (CMSA), the examining board for the national exit examinations for registrars (SA postgraduate medical trainees), specifies five-unit standards and 83 programmatic learning outcomes to be achieved at the end of a 4-year specialist training programme.^30,31,32 The national LP was designed to capture the evidence for 50 of the 83 learning outcomes.^31,33 The LP records evidence of registrars’ personal and professional development through reflection and from WPBA tools.³⁰ Detailed learning plans should specify registrars’ learning needs, plans to address their needs and the goals they have achieved.³⁴ Registrars have to obtain a subminimum of 60% in the LP for each of 3 years to progress to the following year and qualify for the national exit examinations.³⁰

The guidelines for the national LP recommend establishing effective supervisor-registrar relationships with regular engagements, including monthly meetings, at least 6 h of dedicated weekly training and supervision and adequate mentoring.³⁰ Learning portfolios can be handwritten or electronic, depending on university requirements, but electronic portfolios are associated with better completion rates, improved quality of scoring entries, increased frequency of use and sustained supervisor-registrar engagement.³⁵ Previous SA studies found that registrars’ reflective skills, knowledge of the requirements, good supervisor feedback and regular engagement with supervisors positively influenced LP use.³⁴

The skills list for registrars was revised and validated in a 2018 national Delphi study.³⁶ This list of 205 core and 39 elective skills is part of the national LP requirements.³⁰

There are four self-reported competence levels for skills:

There are few studies on WPBA and using LPs in postgraduate medical training in low-middle countries, especially in public district health systems where WBL occurs across various primary health care (PHC) facilities and hospitals. This study adds to the discourse by quantifying evidence of learning in LPs, evaluating the AfL and AoL, and investigating the influence of context, such as the training district or the year of training, on the nature of assessments.

Family medicine was recognised nationally as a new speciality in 2007. The University of the Witwatersrand (Wits University) introduced a full-time postgraduate decentralised FM training programme in 2008. The training programme is structured across 4 years of training. Registrars undergo 2- to 3-week theoretical training blocks at the university annually, depending on the training year. The university also provides an online component comprising FM-related clinical and non-clinical topics, research and journal clubs. Fourth-year registrars focus on the research component and undergo elective rotations to gain further knowledge in a clinical discipline or an area, depending on the registrar’s interest gained during training years.

The FM registrar training programme is based in PHC clinics, community health centres (CHC) or district hospitals across the five study districts. Community health centres are 24-h nurse-led facilities supported by general practitioners and family physicians (FPs). District hospitals are 50–300 bedded hospitals serving a defined population and providing 24-h comprehensive care services and run by general practitioners with a few general specialists, including FPs.³⁷ Registrars rotate every 2 to 3 months in the district or regional-hospital departments for skills training. Skills rotations occur with registrars rotating through various departments, including internal medicine, obstetrics and gynaecology and paediatrics. The timing, location and duration of clinical rotations vary between districts. Decentralised clinical training also includes weekly training sessions at district regional training centres comprising academic discussions (clinical and non-clinical topics), clinical case discussions (one-on-one or group) and WPBAs using the mini-clinical evaluation exercise (mini-CEX) or direct observation of procedural skill tool. The structure of these sessions varied slightly across districts. Another training component is the monthly or bimonthly one-on-one engagements, captured in the LP, between registrars and supervisors to discuss registrars’ learning needs, review their learning plans and progress and identify gaps to be addressed. In some districts, registrars also have observed consultations or procedures with FP supervision at CHCs or PHCs.

The University of the Witwatersrand FM registrar training uses the programmatic assessment model,³⁸ with multiple assessments during the training year. Assessments for learning include WPBAs across the CMSA-prescribed curriculum captured in the LP. Other LP sections are learning plans, educational meetings, direct observations, written assignments, logbooks, certificates for attending courses, meetings and end-of-year assessments.³⁰ Assessment for learning is used to decide annual registrar progression: the quarterly assessments (QA) and the LP overall score are considered. The QA are observed consultations or procedures evaluated using WPBA tools by a supervisor from another district. The QA scores were taken as AfL separately from the overall LP scores contributing to the year mark. The involvement of multiple supervisors improves the validity of WPBA.²⁴ Assessment of learning includes university examinations conducted 18 months into the training programme and the national exit examination after at least three training years as the final high-stake assessment towards specialist qualification.

In the logbook section of the Wits University FM LP, registrars must achieve competencies clustered into core and elective skills within 14 clinical domains aligned to CMSA requirements. These clinical domains are adult health, ear, nose and throat (ENT), eyes and skin, women’s health, emergency care, child health, consultation skills, clinical governance, orthopaedics, anaesthetics, forensics, community-oriented primary care (COPC), clinical administration, teaching and learning and palliative care. Registrars must achieve competence at level D for all core skills and level C or D for elective skills. Their immediate supervisors evaluate registrars’ bi-annual self-assessments of skills in their logbooks before submitting the LPs to the university for final evaluation by the portfolio committee.

Six LPs from each year of registrar training were assessed (N = 18). The total LP median scores and interquartile ranges were 76.3 (62.1–81.2), and the median and interquartile ranges for the QA scores were 64.9 (60.7–69.5).

The median and interquartile ranges calculated for the total LP scores and various LP sectional and QA scores across the training years (Y1–Y3) are shown in Table 1. The total LP median scores were higher in Year 3 compared to Year one (Y1) and Year two (Y2) (Table 1). The Year two registrar’s QA median scores were higher than Years one (Y1) and three (Y3) (Table 1). Supervisors awarded full marks (10/10) in some LP sections in Y1 and Y2 but not in Y3 (Table 1).

Table 2 shows the median and interquartile ranges calculated for the total LP scores and various LP sectional and QA scores across the five districts (D1–D5). The total LP median scores varied considerably across D1–D5 (Table 2), but not the QA median scores. Again, some LP sections (learning plans, educational meetings and observations) had median scores of 10/10 in some districts (Table 2).

In the first- and second-year summative assessments (AoL), 6 out of 12 registrars passed the 18-month university examinations on the first attempt; five of the 12 failed them, and one registrar did not take the mid-point examinations. Of the six registrars who completed the 3 years of training, four who sat the national exit examination passed on their first attempt, and two did not take the examination.

The logbook skills section was incomplete across all training years, but primarily in the first year: four first-year registrars did not complete the section. The self-assessed competence levels of 18 registrars for 205 core clinical skills and 39 elective skills varied considerably (Appendix 1). Depending on the registrars’ self-assessment and the type of skill, competence levels of skills varied and scored from A to D (Appendix 1). Third-year registrars self-assessed their competence as higher (D scores) in most skills than in the other 2 years. The registrars reported higher competence levels in clinical domains such as adult health, women’s health, child health and emergency care than ENT, eyes and skin, orthopaedics and anaesthetics. First- and second-year registrars scored low in domains like orthopaedics, anaesthetics, ENT, eyes and skin because these clinical rotations occur during their third year of training. Most registrars, even first-year registrars, reported higher competence levels in consultation, clinical administration, clinical governance, COPC and teaching and learning domains (Appendix 1).

Although the self-reported level of competence in performing clinical skills was relatively better among third-year registrars compared to Years one and two, some core skills still included levels A, B and C. Examples included a laparotomy for ectopic pregnancy, cricothyroidotomy, assisted vaginal delivery, proctoscopy, vasectomy and reduction of elbow dislocation (Table 3). The total number of registrars in each year was six but varied as registrars did not report on many core skills and left them incomplete (Table 3).

Table 4 shows the total skills set score medians and interquartile ranges for all 14 clinical domains in the LP across the 3 years of training. These medians were higher in adult health, women’s health, emergency care and child health for second- and third-year registrars than ENT, eyes and skin, orthopaedics and anaesthetics domains (Table 4). The Year two median was higher than Year three for the COPC and teaching and learning domains. When total core skills median scores progressed across the training years, elective skills median scores remained low across all three training years (Table 4).

This study examined the evidence of registrar learning from scores and self-reported clinical-skills competence levels recorded in LPs and QAs. The major finding was that LPs are not being used optimally as self-directed learning tools in decentralised training contexts, as has been found previously.^13,17,42 The scores in various LP sections and skills level scores (for each skill and skill set) were variable in different clinical domains across training years and lacked adequate evidence of registrar progression in knowledge and skills. Although the LPs showed evidence of registrar learning, they can be used more effectively in WPBA.

Multiple assessments from various sources in authentic contexts improve assessment validity and reliability.²² Although there was evidence that the programme included multiple WPBAs, there was also evidence of lenient scoring in LP sections like the learning plans and skills competence levels. The negative impact of lenient scoring may be exacerbated by inadequate narrative feedback, as previously reported for this training programme.³⁵ These combined problems raise concerns about whether LPs are being used effectively to support self-directed learning and suggest that supervisors need more clarity on using LPs in postgraduate training.^13,17

The total LP scores improved across training years, but in QA scores across 3 years, second-year registrars had a higher score average than third-year registrars, similar to previous studies that detailed the inadequacies and leniency of supervisors while scoring WPBAs.⁴³ Other LP sections, such as educational meetings and direct observations, showed the required number of educational meetings and WPBAs between supervisors and registrars. As stated in previous studies,^28,34,42 registrars and supervisors in this study focused on completing LP sections rather than effectively utilising them as a learning tool. Whether supervisor–registrar engagement was quantified or engagement took place according to the expected standards with quality feedback to the registrars needs further exploration. In our WBL context, all LP scores were allocated by immediate FP supervisors and later verified by the university’s portfolio committee. The LP introduction in the first year, with adequate registrar training on its use and ongoing mentoring by supervisors, could have enhanced LP utilisation during self-directed learning.^21,44

Quarterly assessment scores, part of the AfL component, contributed separately from LP scores to decide yearly registrar progression. In authentic clinical settings, supervisors and registrars struggle to use WPBA tools such as mini-CEX strictly as a summative tool during QA, always adding a formative component to those assessments. Similar challenges were identified related to mini-CEX use in high-income countries.^24,43

The self-assessed competence in this study ranged across all four levels. Self-assessment is a feasible method for registrars to report their perceived competence, but it is susceptible to over- or under-reporting.^45,46 Although the logbooks ensure that trainees perform the maximum number of skills required for competence,²³ incomplete entries across all three training years undermine that intention. Registrars may have been reluctant to complete the logbook skills section because they perceived their skills competence as inadequate. Alternatively, they may not have felt confident about reporting their competence levels because of inadequate exposure to these skills in their context. Despite the increasing emphasis on self-assessment and critical analysis of trainees’ performance in competency-based medical education,⁴⁷ the registrars appeared to lack the skills to self-assess their performance. The reported competence levels improved in higher training years, suggesting continuous skills learning at their workplaces. Registrars perform skill rotations in major disciplines during their first and second years of training, allowing them to acquire all needed skills. It was encouraging to find that registrars reported adequate competence levels in domain-specific skills learnt earlier in their training, such as consultation, clinical governance and administration, community-oriented primary care and teaching and learning.

A concerning finding was that specific skills such as cricothyroidotomy, assisted vaginal delivery, laparotomy for ruptured ectopic pregnancy and reducing an elbow dislocation were some examples of skills in which third-year registrars reported lower skills competence levels. Third-year registrars should be fully entrusted to perform these essential skills and be able to train junior registrars or students. Registrars’ skills deficiencies should be included in learning plans as learning needs, discussed with supervisors and planned on how to acquire them. Achieving adequate procedural skill competence levels by registrars is a priority in full-time postgraduate FM training programmes compared to older part-time programmes.⁴⁴ The medical officers working in district hospitals in similar contexts reported adequate competence in performing these procedures.⁴⁵ Once qualified as an FP, the registrar will need to function as a consultant and capacitator for medical practitioners and other health workers by performing various surgical and obstetric skills to strengthen district hospital services.⁴⁸ This study reiterated that registrars require adequate exposure to learning opportunities to achieve mastery in performing core procedures encountered commonly in primary-care settings.

Third-year registrars reported lower competence levels on core clinical skills such as proctoscopy, applying clubfoot cast, vasectomy, culdocentesis and performing a brachial block. Questions that need to be answered include whether registrars had sufficient opportunities or allocated time to practise these skills during clinical rotations and whether these procedures were performed in sufficient numbers in various disciplines of the regional and district hospitals where registrars rotated. Current trends in medical education emphasise contextualising the curriculum and learning opportunities to acquire knowledge and skills needed in the health care system where trainees practice.⁴⁹ While the FM skills list was revised in 2017,³⁶ some core skills are still not performed sufficiently at peri-urban district hospitals.^45,50 Perhaps it is time to revise the FM registrar training skills list to include more relevant context-specific procedural skills.

The impact of the training context on achieving the required levels of core skills competence may not be sufficiently considered. Family physicians in SA work in various health-system contexts like district, regional and tertiary hospitals, private general practice or rural and urban clinics, where they require most of these skills.⁵¹ Mastering all core skills could be more relevant for an FP practising in a rural district hospital where acute health worker shortages and referral challenges are experienced than in urban settings. It may be time to consider separating skill sets for FPs practising in rural and urban contexts.^45,52 Even in high-income countries, doctors working in rural areas reported higher skill levels than their urban counterparts.^46,53 A compulsory rural block for registrars working in peri-urban district hospitals to achieve specific skills, for example, a 2 to 3-month rotation or even a year of longitudinal clinical work from a peri-urban district hospital to a rural district hospital, could potentially mitigate the skills gaps identified among registrars.

Although studies were conducted when the LP was introduced into postgraduate FM training in SA about 10 years ago,⁴² there has not been further research to measure its longitudinal impact. This study initially sought evidence for WBL in the short term, but comparing the findings to studies from a decade ago highlights the long-term impact after LPs were introduced whether they were effectively utilised in WBL. Despite the positive impact of LPs on WBL, this study highlighted several issues that adversely affected how effectively registrars used the LPs, including incomplete logbook sections and a lack of clarity about the roles of FPs.

Workplace-based assessments captured in the LP provide trainees with more opportunities to reflect on their practice in authentic settings, positively influencing learning behaviour.⁴³ Teamwork, professionalism and self-appraisal are all assessed when a mini-CEX tool is used in WPBAs.²² The assessment of complex tasks relied on the trainee’s ability to integrate cognitive, psychomotor and affective components, best evaluated in authentic clinical settings.²² Currently, the LP primarily focuses on scoring systems to determine whether the numbers needed are met or whether registrars achieved adequate skill competence levels. Providing more qualitative feedback on registrar performance will enhance the comprehensiveness of WPBAs. Most importantly, creating a learning environment that encourages a reflective dialogue between trainees and their supervisors is vital for effective LP utilisation.²⁰

This study aimed to evaluate evidence of learning in LPs, formative and summative WPBAs and the influence of the training district and the year of training on assessments. While the findings provided a holistic view of WPBA in FM training in one setting, they could apply to similar contexts at South African universities. Future research on WPBA across multiple training programmes will give a complete picture of postgraduate FM training in SA, which may also benefit other sub-Saharan African countries.