The national portfolio for postgraduate family medicine training in South Africa: a descriptive study of acceptability, educational impact, and usefulness for assessment

Background Primary care providers should be competent in brief behaviour change counselling (BBCC). A new model of BBCC was developed in South Africa. Tools are needed for training and research to evaluate BBCC. Aim To evaluate the validity and reliability of a tool to assess BBCC. Setting Primary care providers in Western Cape, South Africa. Methods Exploratory sequential mixed methods included initial qualitative feedback from an expert panel to assess validity, followed by quantitative analysis of internal consistency, inter- and intra-rater reliability. Six raters assessed 33 randomly selected audiotapes from a repository of 123 tapes of BBCC at baseline and 1 month later. Results Changes to the existing tool involved item changes, added items and grammatical as well as layout changes. The ‘Assessment of Brief Behavioural Change Counselling’ tool (ABC tool) had good overall internal consistency (Cronbach’s alpha 0.955), inter-rater (intra-class correlation coefficient [ICC] 0.813 at follow-up) and intra-rater reliability (Pearson’s correlation 0.899 and p < 0.001). Sub-scores for the Assist (ICC 0.784) and Arrange (ICC 0.704) stages had lower inter-rater reliability than the sub-scores for Ask (ICC 0.920), Alert (ICC 0.925) and Assess (ICC 0.931) stages. Conclusion The ABC tool is sufficiently reliable for the assessment of BBCC. Minor revisions may further improve the reliability of the tool, particularly for the sub-scores measuring Assist and Arrange. The ABC tool can be used in clinical training or research studies to assess fidelity to this model of BBCC.

show abstract

“…For registrars in family medicine, it can also be included in workplace-based assessment and their portfolio of learning. 27 …”

Section: Discussionmentioning

confidence: 99%

The psychometric properties of a tool to assess brief behaviour change counselling in South Africa

Fouche

Mash

Malan

2020

African Journal of Primary Health Care & Family Medicine

View full text Add to dashboard Cite

show abstract

“…Feedback on which sections of the portfolio should be included for summative assessment was requested from registrars, supervisors and programme managers in a national survey. 38 From this feedback a core group of experts designed the PAT to score each of these sections and added a final global assessment for the whole portfolio. The draft PAT was also discussed with the eight heads of family medicine departments in the country.…”

Section: Research Methods and Designmentioning

confidence: 99%

Reliability testing of a portfolio assessment tool for postgraduate family medicine training in South Africa

Jenkins

Mash

Derese

2013

Afr. j. prim. health care fam. med.

Self Cite

View full text Add to dashboard Cite

BackgroundCompetency-based education and the validity and reliability of workplace-based assessment of postgraduate trainees have received increasing attention worldwide. Family medicine was recognised as a speciality in South Africa six years ago and a satisfactory portfolio of learning is a prerequisite to sit the national exit exam. A massive scaling up of the number of family physicians is needed in order to meet the health needs of the country.AimThe aim of this study was to develop a reliable, robust and feasible portfolio assessment tool (PAT) for South Africa.MethodsSix raters each rated nine portfolios from the Stellenbosch University programme, using the PAT, to test for inter-rater reliability. This rating was repeated three months later to determine test–retest reliability. Following initial analysis and feedback the PAT was modified and the inter-rater reliability again assessed on nine new portfolios. An acceptable intra-class correlation was considered to be > 0.80.ResultsThe total score was found to be reliable, with a coefficient of 0.92. For test–retest reliability, the difference in mean total score was 1.7%, which was not statistically significant. Amongst the subsections, only assessment of the educational meetings and the logbook showed reliability coefficients > 0.80.ConclusionThis was the first attempt to develop a reliable, robust and feasible national portfolio assessment tool to assess postgraduate family medicine training in the South African context. The tool was reliable for the total score, but the low reliability of several sections in the PAT helped us to develop 12 recommendations regarding the use of the portfolio, the design of the PAT and the training of raters.

show abstract

“…The shift of focus to workplace-based assessments by means of a portfolio of learning coupled with increasingly better access to the internet and use of web-based educational platforms for distance education (e.g. Moodle, Blackboard) enabled the development of an e-portfolio [12, 13]. Some programmes are utilising the on-line course management software, while others are looking at software designed specifically for portfolios of learning.…”

Section: Introductionmentioning

confidence: 99%

Implementing and evaluating an e-portfolio for postgraduate family medicine training in the Western Cape, South Africa

et al. 2019

View full text Add to dashboard Cite

Background In South Africa it is compulsory to submit a satisfactory portfolio of learning to gain entrance to the national exit examination of the College of Family Physicians and to qualify as a family physician. A paper-based portfolio has been implemented thus far and the need for an electronic portfolio (e-portfolio) was identified. The aim of the study was to describe and evaluate the implementation of an e-portfolio for the training of family medicine registrars in the Western Cape province of South Africa. Methods Mixed methods were used. A quasi-experimental study evaluated paper- and e-portfolios from the same 28 registrars in 2015 compared to 2016. Semi-structured interviews were conducted with 11 registrars or supervisors to explore their experiences of using the e-portfolio. Quantitative data was analysed in the Statistical Package for Social Sciences and qualitative data in Atlas.ti. Results Most respondents found the e-portfolio easier to use and more accessible. It made progress easier to monitor and provided sufficient evidence of learning. Feedback was made easier and more explicit. There were concerns regarding face-to-face feedback being negatively affected. It was suggested to have a feedback template to further improve feedback. Several aspects were significantly better in the e-portfolio such as feedback on the registrar’s general behaviour, alignment with learning outcomes, less feedback based on hearsay and acknowledgement of the feedback by the registrar. Although not statistically significant, there was an increase in the usage of the e-portfolio, compared to the paper portfolio. Conclusion In general, the e-portfolio is an improvement on the paper-based portfolio. It is easier to access, more user-friendly and less cumbersome. It makes feedback and monitoring of progress and development of registrars easier and more visible and provides sufficient evidence of learning. Its implementation throughout South Africa is recommended. Electronic supplementary material The online version of this article (10.1186/s12909-019-1692-x) contains supplementary material, which is available to authorized users.

show abstract

The national portfolio for postgraduate family medicine training in South Africa: a descriptive study of acceptability, educational impact, and usefulness for assessment

Cited by 22 publications

References 54 publications

The psychometric properties of a tool to assess brief behaviour change counselling in South Africa

The psychometric properties of a tool to assess brief behaviour change counselling in South Africa

Reliability testing of a portfolio assessment tool for postgraduate family medicine training in South Africa

Implementing and evaluating an e-portfolio for postgraduate family medicine training in the Western Cape, South Africa

Contact Info

Product

Resources

About