According to a review by Davidson and Kasznjak, the core components of mindfulness are attention, awareness, memory, and discernment.¹⁰ However, Voci et al. argue that mindfulness should not be viewed as an analytical awareness but rather as an attentiveness imbued with a kind, non-judgemental attitude towards internal and external experiences.¹¹ This form of awareness allows one to fully accept present internal and external experiences without the need to change them.

By evaluating the items present in available open-source mindfulness questionnaires,^12,13,14 it was found that the items in the MPCCQ mindfulness subfactor reflect both the attention and awareness facet described by Davidson and Kasznjak,¹⁰ as well as the non-judgemental observing dimension outlined by Voci.¹¹ Considering the given information, the definitions of the mindfulness subfactors were slightly revised from the original author’s definition to clarify this distinction⁷:

Mier et al. situate emotion recognition within the framework of intention recognition or Theory of Mind (ToM), which is a component of the broader study of social cognition.¹⁵ This perspective suggests that recognising others’ emotions helps us to understand their intentions. Notably, emotion recognition develops earlier than the ability to understand and interpret emotions, implying that it is an unconscious process.¹⁶ This is reflected in Coricelli’s ToM model, which consists of unconscious automatic processes – such as emotion recognition, emotional contagion and action recognition – followed by conscious processes such as probing or hypothesis testing, which ultimately lead to a decision about a person’s intentions.¹⁷

It is theorised that the items in the MPCCQ assess emotion recognition along with both conscious and unconscious processes, as described. The two extracted factors were labelled as recognition and labelling of emotion (factor 1) and emotional probing (factor 2). The definition of the two extracted factors remains unchanged⁷:

In social psychology, compassionate actions are often classified under and used in place of the larger construct called pro-social behaviours. Pro-social behaviour can be defined as a voluntary behaviour intended to benefit another.¹⁸ This broad construct encapsulates a wide variety of related helping behaviours that Dunfield and Gross et al. categorise into three subtypes according to their function.^19,20 The three subtypes include helping, sharing, and comforting, which function to provide instrumental needs, provide for unmet material desires, and ease emotional distress, respectively. This model of pro-social behaviours is used to explain the factor fission found in the compassion action orientation subscale of the MPCCQ.⁷

To elaborate, the items loading onto the first subfactor correspond to Dunfield’s comforting subfactor in pro-social behaviours where the aim is to relieve emotional distress.¹⁹ The other subfactor that emerged corresponds with Dunfield’s helping action intended to aid instrumental needs and was labelled as organising resources.¹⁹ Formally, the two subfactors are defined as follows⁷:

The literature review clarified the connotative meanings of the subfactors mindfulness, emotion recognition, and compassion action orientation, providing a theoretical basis for refining the MPCCQ measurement model. However, the extent to which these subfactors function as distinct constructs remained unclear and were subsequently examined by applying the Schmid-Leiman transformation to the completely standardised second-order factor measurement model parameter estimate data obtained by Visser.²¹ Results indicated that a general factor representing the variance shared by the extracted first-order factors accounted for over 82% of the explained variance in item responses for all three subscales. This suggests that, while the connotative meaning of the subfactors may be theoretically and practically important, the explained subscale item variance was largely because of a general factor rather than the residualised first-order factors representing the themes unique to the extracted first-order factors. Accordingly, it was deemed more appropriate to refine the structural model using a bifactor structure, wherein a general factor accounts for the majority of explained item variance, alongside two uncorrelated residualised factors that manifest only in specific items. The refined structural model is presented in Figure 1.

The new structural model includes additional factors, and the original questionnaire developed by Visser does not have enough items to accurately measure the latent constructs that emerged from the factor fission as conceptualised in Figure 1. Consequently, the study followed Visser’s recommendation to develop three to four additional test items per subfactor to ensure there are at least five items per subscale. The additional items were specifically developed to tap into the themes unique to the extracted first-order factors rather than the (general) theme shared by the first-order factors. Given the theoretical background, and the information available from the original study by Visser, the researchers decided to employ both inductive and deductive approaches in item development. The literature review guided the constructs each item should address (deductive item development), while transcripts from interviews conducted by Visser provided the specific contextual content for each item (inductive item development).

As per best practice guidelines, the refined questionnaire should be reviewed by a small group of subject matter experts (SMEs) to assess the relevance of the items and the overall suitability of the questionnaire.^22,23 The following section details the research methodology used to evaluate the refined MPCCQ.

The objective of the qualitative review is to revise the MPCCQ items in accordance with the updated structural model informed by the literature review. This is achieved by:

The most appropriate way to achieve the above objectives is through expert reviews by SMEs. Given that the data involves expert opinion, a qualitative research method is the most appropriate. Qualitative methods are used to gather rich, detailed data that allow for a flexible and thorough exploration of each participant’s perspectives. To enhance the rigour of the research, participants were also asked to rate the items following the approach used by Lawshe and Nel.^24,25 Integrating a quantitative approach for data triangulation aligns with the post-positivist paradigm of measuring objective reality, ensuring that decisions regarding item revisions are not solely based on the subjective interpretation of qualitative data, but are supported by accurately quantifying participants’ opinions.

Subject matter experts were asked to rate each item on the revised subscales of the mindfulness, emotion recognition, and compassion action orientation based on their clarity and validity on a scale from 1 to 3, as shown in Table 1.

If the participants rated any item below three for clarity or validity, they were requested to provide open-ended comments about the items, whereafter the researcher altered the item content based on these suggestions depending on the level of agreement between SMEs. The level of agreement between SMEs was evaluated using an aggregate score of their ratings, called the content validity ratio (CVR) of an item, which ranges from 1 to +1.²⁵

A value of 1 indicates that all SMEs found the item clear and/or valid; a value of 0 means half of the SMEs agreed on its clarity and/or validity; and a value of –1 suggests that all SMEs found the item unclear and/or invalid. In this study, a CVR value below 0.54 indicates statistically significant disapproval (one-tailed, p < 0.05) by SMEs with regard to the appropriateness of the item, suggesting that alterations to the item are necessary.²⁵ The item was rewritten if either the clarity or validity CVR reflected problems with the item. It is important to note that the goal was not to achieve complete agreement among SMEs, but rather to revise items that received unanimous disapproval from them, as evaluated by the CVR.

However, slight modifications were made to items to eliminate any potential confusion about what was being asked, even if the CVR was above the cut-off value. However, in those cases, care was taken to ensure that these modifications did not alter the content of the items. Furthermore, if the item was deemed to be repetitive by any participant, then the item was rewritten. Figure 2 summarises the item alteration process.

The SMEs were not asked to re-evaluate the new and revised items because of time constraints and concerns about overburdening participants. Ideally, item revision should continue until a consensus is reached among SMEs. Nevertheless, the researchers carefully implemented the SMEs’ feedback, taking care to avoid introducing their own biases into the items.

Purposive and snowball sampling were deemed the most appropriate methods for participant selection. After the selected SMEs were approached to participate in the study, they were asked to refer other relevant individuals in their field to participate. The researchers aimed to recruit a minimum of five SMEs to participate in the study as this allows for the implementation of Lawshe’s CVRs.²⁵ However, a larger sample size enhances the confidence with which the results can be considered genuinely reflective of expert opinions; consequently, the researchers encouraged and allowed a greater number of participants to participate in the study. By the end of the data-gathering phase, 14 SMEs had completed the review, and this was deemed sufficient.

While the target population for the MPCCQ is medical practitioners working in the public healthcare sector in South Africa, including experts in fields related to questionnaire development such as psychometry, industrial psychology, and language editing may increase the chances of identifying problematic items that may cause confusion because of possible ambiguous or compounded item presentation. This research, therefore, had the following inclusion criteria:

Table 2 provides a detailed description of the sample characteristics and shows that the participants varied greatly in terms of their years of experience, ranging from three years to 40 years, with half of the participants (n = 8) having between three years and 10 years of experience. Furthermore, nine of the 14 participants’ home language was Afrikaans, four participants’ home language was English, and one participant did not specify. The majority of the sample (seven participants) were from family medicine, two participants were from industrial psychology, and there was one participant each from pathology, psychiatry, psychometry, language editing, and obstetrics and gynaecology. Two participants were with specialisations in the fields of psychiatry and pathology, therefore, they fell outside the inclusion criteria. Despite this, they were included in the final data analysis because the pathologist had extensive experience working in South Africa’s public healthcare sector and the psychiatrist had substantial knowledge within the field of psychology. Thus, the final sample comprised 71% medical practitioners and 29% specialists in test development.

In this study, the researcher provided the full response of each participant in Online Appendix 1. The participants’ views are therefore documented accurately and transparently. In addition, the Lawshe rating method was employed alongside the qualitative feedback as a triangulation method whereby the participants’ views were both recorded and quantified.

In the context of this study, confirmability is achieved when the researcher can demonstrate that the alterations made to the item content are substantively derived from the qualitative data. This study took the following steps to ensure confirmability:

It is generally difficult in qualitative research to demonstrate that findings can be applied to other contexts, and as such, it is often the case that the researcher specifies the cases to which the results can be applied. This is carried out by documenting factors such as sample composition, the number of participants involved in the research, the data collection methods employed, and the time period over which the data were collected.²⁶ These characteristics are observed in this study.

To ensure dependability, every step of the analysis was thoroughly documented, and the methodology was explained in detail. A table meticulously documenting each alteration to the items can be consulted in Online Appendix 1. Consequently, future researchers should be able to replicate the study documented here and assess whether this study adhered to proper research practices and protocols.

An application for full ethical approval was made to the Social, Behavioral and Education Research Ethics Committee (REC: SBE), Stellenbosch University and ethics consent was received on 15 February 2024. The ethics approval number is 27193.

To protect the participants’ anonymity, they were not required to record identifiers on their questionnaires, which were instead assigned numbers that could not be traced back to them and sensitive demographic information unrelated to the evaluation of the questionnaire’s content was not requested. However, to ensure scientific rigour and allow for the replication of the study by other researchers, the following demographic information that might influence the SMEs’ ratings of the scale was collected:

Written consent was obtained by means of an informed consent form. This form explained the study’s aim, nature, procedure, benefits, confidentiality, and complaint procedures. The informed consent form was emailed to the SMEs along with the questionnaire. Stellenbosch’s Social, Behavioural and Education Research Council approved the study on 15 February 2024 (Project number 27193).

When analysing the qualitative feedback from participants, it became clear that their educational backgrounds influenced the content of their feedback. The psychometrist and language editor were particularly thorough in scrutinising sentence structure, readability, and the function of the wording. For instance, the psychometrist made an insightful observation about the value of the phrase ‘as time does not allow for this’ in item E1 aimed at measuring emotion recognition:

Indeed, the phrase, ‘as time does not allow for this’ may be more appropriate for the factor compassionate action, because it is often the case that medical practitioners view short consultation times as an unavoidable consequence when they are required to attend to numerous patients per day, and this is not necessarily an indication of emotion recognition.⁹ Similarly, the language editor’s detailed understanding of the English language is evident in her feedback on item M12:

In contrast to the feedback from the language and psychometry specialists, the medical professionals were less focused on the precise wording of the items and more concerned with their applicability to their specific context. Based on their personal experiences, they found that some items might make unreasonable demands on medical practitioners. For example, consider the feedback from the medical professionals P4, P5, and P6 on item M7 regarding note-taking being perceived as a distraction:

As an additional example, note how medical practitioners highlighted potential confusion with the phrase ‘resistance to treatment’. While industrial psychologists and psychometrists naturally interpreted the phrase as referring to behavioural or emotional resistance, the doctors pointed out that it can be understood as biological resistance:

The rich qualitative data collected by the research participants indicate that it was a fruitful exercise to include SMEs from various educational backgrounds to review the questionnaire. As a whole, they were able to provide feedback on both technical and practical issues with the items.

When more narrowly defined constructs are used, important information relating to the general construct may be overlooked. Conversely, when more general items are used, it makes it more difficult to provide differential feedback on specific competencies that require improvement.²⁷ Participant P5 was particularly perceptive to this and raised concerns that focusing on emotion recognition may exclude important considerations that influence patient care. Take the following comments as an example:

While this is valuable input, generalising the scale to include broader aspects of intention recognition as opposed to emotion recognition may ultimately result in the multidimensional construct collapsing into a general construct. In other words, differential feedback on emotion recognition specifically will not be possible if the item is more generalised. The items were therefore left unchanged.

That said, an additional, and very important point highlighted by participant P5, is that the narrowly defined items may be too similar. Because the facet being measured is so narrowly defined, creating items with a clear distinction from one another can be difficult. For participant P5, this was particularly true for the facet of attention and awareness. Similarly, participant P11 found that the items probing emotion recognition competencies may be too similar and fail in their task to extract unique information:

Given the danger of asking the same question more than once, items M7, M8, E1, and E4 were rewritten to better differentiate themselves from other items. The final aspect highlighted by the SMEs was their overall opinion of the questionnaire. Their feedback primarily focused on two themes: the length of the questionnaire and its overall utility. These themes are discussed in the following sections.

It is important to acknowledge the time it took respondents to complete the questionnaire, as this can have an impact on its practical administration. Specifically, the questionnaire’s length may affect the response completion rate, the viability of re-assessments, the user experience, and the productivity of employees who need to take time off work to complete it (when used in an HR context). Three of the SMEs expressed concerns about the questionnaire’s length, as highlighted by the following quotes:

The responses indicate that the questionnaire may take longer to complete for individuals whose home language is not English. In addition, although the items need to be very specific, this specificity can make them cumbersome to read and sometimes feel long-winded to the participants. This feedback is particularly concerning because participants were only asked to evaluate 9 of the 12 factors. Thus, it may take participants up to an hour or more to complete the full questionnaire, raising concerns about its practicality and frequency of use among healthcare professionals who are often overworked and under pressure.

Despite concerns about the questionnaire’s length and verbosity potentially affecting its administration and completion rate, respondents still found the content insightful. In fact, some respondents noticed that completing the questionnaire was personally enriching, allowing them to critically reflect on their behaviours in ways they had not performed before. Take the below quotations as an example:

From the feedback provided, it can be concluded that the questionnaire potentially achieves its purpose of promoting self-reflection.

The sample comprised participants mostly from family medicine, and most of them were native speakers of Afrikaans or English. Consequently, the perspectives of practitioners in paediatrics, internal medicine, and surgery – as well as those whose home language is not Afrikaans or English – were not included in this study. This study did not administer the questionnaire and the reliability and validity of the new measurement model were therefore not tested.

It is recommended that the revised questionnaire be administered to medical practitioners in South Africa in order to properly validate the questionnaire. To validate the structural and measurement model of the MPCCQ, the researcher should use well-established techniques such as exploratory and confirmatory factor analysis, along with structural equation modelling. It is acknowledged that in the validation of the revised MPCCQ, the operationalisation of the six narrow factors related to mindfulness, recognition of emotions, and compassion action orientation will present an interesting methodological challenge. Moreover, obtaining scores on these narrow (residualised) mindfulness, recognition of emotions, and compassion action orientation factors to unlock the practical value of formative feedback for medical practitioners on the themes unique to the extracted first-order factors will present an even more exciting methodological challenge.