A 360 DEGREE ASSESSMENT OF PROFESSIONALISM AMONG RESIDENTS IN A TERTIARY CARE HOSPITAL.

Professionalism is a core skill in medicine and practice and it includes a combination of values, behaviors, and relationships which form the basis of trust that the population places in doctors [1]. It is among the fundamental areas that accrediting organizations like the Accreditation Council on Graduate Medical Education (ACGME) and the Medical Council of India (now NMC) have outlined the significance of during residency education [2,3]. Nevertheless, it has got a complicated and dynamic problem of defining, teaching and assessing professionalism [4].

According to professionalism, there is accountability, altruism, duty, excellence, honor, integrity, and respect towards others [5]. Although it is important, the evaluation of professionalism is usually based on subjectivity, inconsistency, and faculty-based, which can lack a comprehensive perspective [6]. This explains why a holistic strategy like the 360-degree feedback method which incorporates the information of several people like the faculty, fellow workers, nurses, ancillary workers, and patients is necessary [7].

The 360-degree feedback has also become a valid and reliable method of evaluating non-technical skills like teamwork, communication and professionalism [8]. It is more inclusive, less biased, and it encourages trainees to reflect and learn about themselves [9]. Multi-source feedback has been found to be helpful in the behavioral change and professional growth of residents and medical students in a number of studies [10,11].

Moreover, the culture and workplace environment are becoming increasingly understood to have a great impact on professionalism. Therefore, it becomes critical to assess professional behavior of residents in the actual clinical environment using a series of inputs of observers [12]. The application of structured professionalism assessment strategies does not only help in revealing areas in which improvement is necessary but also helps in developing a culture of accountability and constant improvement [13].

In Indian scene, even though medical institutions have started putting professionalism in their curriculum and tests, effective and systematic means of measuring professionalism is yet to be used [14]. It is urgently needed to conduct studies that will discuss the viability of 360-degree assessment and its effectiveness in this cultural and institutional model [15].

It is based on this that the current work was conducted to determine the degree of professionalism among resident doctors in a tertiary care hospital using a 360-degree feedback method. It was intended to check the current level of education and awareness of medical professionalism in residents, to make them aware of the various aspects of professional behavior in clinical practice, and to identify whether the sensitization programme led to a change in their awareness and attitudes to professionalism.

Study Design and Setting

The current study was conducted as a quasi-experimental study in the Bhaskar medical college and hospital in Hyderabad which is a tertiary care teaching hospital. The experiment was carried out within six months. This environment was deemed suitable since the institution offers a formalized academic environment whereby postgraduate residents are engaged in patient care, clinical decision making, and interdisciplinary interaction all through. The study design enabled the evaluation of the professionalism prior to and after the organized educational intervention, thus contributing to the observation of the changes in the professional behaviour and attitudes after sensitization.

Study Population

The population of the study was postgraduate residents who were in a clinical training in Bhaskar Medical College and Hospital. These residents were a suitable sample to be evaluated since they are close to the daily work with patients and cooperate with faculty members, other residents, nursing staff, hospital staff, and patients. The residents in active clinical departments gave a pertinent and valid population to the study since the expression of professionalism in everyday clinical practice is through interpersonal behaviour, responsibility, communication, empathy and ethical behaviour.

Sample Size

The study involved 123 residents. Informed consent had to be provided in writing before all the eligible participants were enrolled. The chosen residents met the inclusion criteria set and were actively deployed in different departments of the clinic at the time of the study. The sample was deemed sufficient in terms of evaluating trends in the field of professionalism and comparing the results of observation before and after the educational intervention in the context of the given institutional environment.

Sampling Technique

The study used a purposive sampling method. The residents were included, and they had to be available in the study period, meet the eligibility criteria, and state that they were willing to participate. The approach was selected since the research was aimed specifically at testing the level of professionalism among the postgraduate trainees who were actively engaged in the working process and could be monitored by various stakeholders in the hospital setting. The sampling strategy was effective to include participants which were pertinent to the study purpose and able to give significant pre-intervention and post-intervention answers.

Inclusion Criteria

Individuals who work in clinical departments were deemed to be eligible in the study. Only willing residents who gave informed consent were recruited. This allowed the participants to be actively engaged in the patient care, as well as contributing to the study process voluntarily.

Exclusion Criteria

The study excluded residents who were on long term leave during the period of the study or those who were posted in non clinical departments. Moreover, those who were not consenting to participation were not included. This was done to ensure that the study population is uniform and also to make sure that only individuals with regular clinical exposure and the opportunity to be assessed on multi-source were considered.

Intervention

To enhance their awareness and knowledge about professionalism, the involved residents were subjected to a well-organized sensitization programme. This learning intervention was aimed at enhancing their understanding of professional values and expected behavior in clinical practice. Case-based scenarios, focused group discussions, lectures, and interactive teaching sessions were the components of the sensitization sessions. Clinical situations were used to make residents connect theoretical concepts of professionalism with the practical situations of patient care. Group discussions helped to reflect, exchange opinions, and think critically, whereas lectures and interaction elements helped create conceptual clarity of the key areas of professionalism. In general, the intervention was designed to raise the amount of knowledge but also to encourage reflective learning and behaviour change in daily medical practice.

Data Collection Tools

A 360-degree feedback questionnaire that is a pre-tested instrument was used to gather data, specifically created to measure the aspect of professionalism among the medical trainees. The tool encompassed eight fundamental areas of professionalism, which include altruism, accountability, excellence, duty, honour and integrity, respect of others, empathy and communication skills with teamwork. These areas were chosen since they are critical areas of professional practice that are required of the medical practitioners. The answers to every question in the questionnaire were measured on a 4-point Likert scale where 1 was considered as poor and 4 as excellent. This scoring pattern enabled the respondents to rate professional behaviour as observed in a systematic and standard way.

Data Collection Method

Professionalism was evaluated by multi-source feedback method otherwise referred to as 360-degree assessment. The feedback was to be provided by various types of persons that were in constant interaction with the residents in the clinical environment and this included the faculty members, peers or co-residents, nursing staff, administrative and support staff and the patients. The reason why this approach was selected is that professionalism is best perceived depending on the observations of more than one viewpoint and not a single assessor. Feedback was also obtained at two stages, prior to the sensitization programme and after the professionalism training session was done. This pre and post intervention assessment assisted in determining the changes in professional behaviour after the educational module.

Data Analysis

The data obtained were coded and put into SPSS version 16.0 to analyze them statistically. The frequency, percentages and average scores were used as descriptive statistical measures to summarize the baseline characteristics and overall response patterns. Comparison analysis was done to investigate the changes in the observations between the pre-test and post-test. Where possible, the Chi-square test was used to compare the categorical distributions of responses to pre and post-intervention. The p-value of below 0.05 was regarded as statistically significant. The statistical test was to be conducted to identify whether the structured sensitization programme had any significant difference in the scores of professionalism in the measured domains.

Ethical Considerations

Ethical approval was sought before the study was conducted among the Institutional Ethical Committee of Bhaskar Medical College. The purpose and nature of the study were explained to all the participants and informed consent was obtained in writing prior to data collection. The involvement was voluntary. The study ensured that there was sufficient care to ensure confidentiality and anonymity of the responses. There was no disclosure of any personal identifiers during the process of data handling or reporting. The research was carried out based on the ethical principles of research involving human participants with respect to the autonomy of participants, their privacy and institutional norms.

General description of the sample under analysis

The number of resident doctors in the analysis was 123. The pre- and post-sensitization programme was evaluated using a 360-degree feedback tool that is structured. Throughout the study period, majority of the ratings fell within the good and excellent category, which suggests that the residents were generally rated favourably even at the baseline. The post-intervention evaluation, however, revealed that there was a more definitive change towards poor and average ratings to more positive responses in a number of domains of professional behaviour.

Change in professionalism domains item-wise

In the case of individual analysis of the eight items of professionalism, the sensitization post-intervention improved most of the domains. Work with nurses and staff, respectful treatment of patients and family members, clarification of the state of the patient with empathy, delegation of duties, and overall professional competence were areas of the strongest improvement. Conversely, there was a small change in interaction with colleagues and a small redistribution of the scores in punctuality. The post-test change in the item on overall professional competence was the most noticeable, with a significant change in the category of good to the category of excellent (Table 1).

Table 1:  Item-wise distribution of pre-test and post-test ratings

The positive percentage rose to 78.45 to 96.73 regarding communication with nurses and staff, 68.46 to 96.74 with regard to respecting the patient and family members, and 72.30 to 95.93 when explaining the condition of the patient in an empathetic manner (Figure 1). General professional competence also increased dramatically as there was positive ratings of 82.30 percent at the baseline and 98.36 percent at the sensitization. Statistically significant post-test changes were implied in approximate distributional comparison of most of these domains, but not in interaction with colleagues.

         Figure 1: Favourable rating of items before and after sensitization

When the eight professionalism domains were analysed collectively, there was a clear improvement in favourable ratings (defined as “good” and “excellent” responses) following the sensitization programme. The mean favourable proportion increased from 80.57% at baseline to 94.91% after intervention, corresponding to a mean paired increase of 14.34 percentage points (95% CI: 6.64 to 22.04). Statistical testing at the item level demonstrated a significant improvement using the Wilcoxon signed-rank test (p = 0.0156), with a large effect size (r = 0.842). A paired t-test yielded similar findings (p = 0.0031) with a substantial standardized effect (Cohen’s dz = 1.556), indicating a strong intervention effect on professionalism scores.

All domains showed improvement in favourable ratings after sensitization, although the magnitude of change varied across domains (Table 2). The largest gains were observed in communication with nurses and staff, respectful interaction with patients and families, explanation with empathy, and overall professional competence. Punctuality in duties demonstrated minimal change compared to other domains, suggesting that certain behavioural attributes may be less responsive to short-term interventions.

Table 2: Item-wise comparison of favourable professionalism ratings before and after sensitization

Source-based comparison of rating

Evaluation of the rater group revealed that the trend of improvement was not identical among the evaluators. Faculty ratings showed a definite post-test change to good and excellent. Nurses also documented improvement especially with an significant increase in excellent ratings indicating the enhancement in the day-to-day professional behaviour following sensitisation. The same favourable trend occurred in the ratings of administrative and support personnel. The change in peer assessment was not as significant as in the other groups, which suggests that the co-residents might have found that there was a smaller behavioural difference between the period of intervention (Table 3).          

Table 3: Pre-test and post-test rating pattern according to rater group

The percentage of faculty positive ratings went up to 94.30% compared to 72.37% and the same percentage among nurses went up to 88.62% compared to 78.46%. The favourable ratings among peers decreased to 82.97 and increased to 84.71, which means that there was a minimal change (Figure 2 & 3). The greatest change of non-faculty assessors was in the other category where favourable ratings rose by 79.09 to 96.67.

          Figure 2: Favourable ratings by rater group before and after sensitization

Figure 3: Slope plot illustrating the direction and magnitude of change in favourable ratings across the eight professionalism domains

Favourable ratings increased across all rater categories following the intervention (Table 4). The mean favourable score rose from 78.22% to 91.08%, corresponding to an average increase of 12.85 percentage points.

However, due to the small number of rater categories (n = 4), the paired statistical comparison did not reach conventional levels of significance (Wilcoxon p = 0.1250; paired t-test p = 0.0624), despite a large effect size (Cohen’s dz = 1.451). Faculty and “other staff” groups showed the most pronounced improvements, whereas peer ratings demonstrated relatively modest change.

Table 4: Comparison of favourable ratings across rater groups

Alteration in the relative Likert distribution

At the aggregate level, poor and average ratings decreased after the intervention, whereas the excellent ratings increased. The average percentage that was assigned to poor ratings dropped to 0.303 in the post-test stage compared to the 1.145 in the pre-test stage. On the same note, the mean category dropped down to 5.28 as compared to 18.26. The category of excellence demonstrated a high increase of 21.43 to 24.48. Even though the good category rose to 70.42 as compared to 59.12, it was not statistically significant in the initial analysis (Table 5). Item-level paired analysis showed a significant reduction in lower-end ratings. The mean “poor” category decreased from 1.15% to 0.30% (p = 0.0312), and “average” responses declined from 18.26% to 5.28% (p = 0.0156). In contrast, the increase in “good” and “excellent” categories, although evident, did not reach statistical significance at the item level (Table 5; Figure 4 & 5).

Table 5: Mean pre-test and post-test distribution across Likert categories

Figure 4:  Mean score of the eight items of the questionnaire with four Likert scales SEM represented by error bars are at item level percentage

Figure 5:  Forest-style plot showing the absolute improvement in favourable ratings for each professionalism

Mean score

The general average Likert score significantly improved in a positive manner and slightly after the sensitization programme. The average score during the pre-test stage was 24.994 and the average score during the post-test stage was 25.123 with a difference of 0.129. Although the absolute change was low, the initial results table indicated that this change is statistically significant with a p-value less than 0.001. This implies that at the aggregate level, the intervention was connected with the quantifiable enhancement of the professionalism profile of the residents (Table 6)

Table 6:  Overall mean score before and after sensitization

On the whole, the results suggest that the 360-degree sensitization exercise was succeeded by the increase in the professionalism ratings in a variety of clinically-relevant areas. The greatest improvements were in communication with nurses and staff, respectful behaviour toward patients and families, empathic explanation of patient condition, discharge of departmental responsibilities and professional competence in general. The post-test trend was not just defined by the fact that favourable ratings increased but also that the number of poor and average ratings decreased. Meanwhile, the extent of the improvement differed among the domains and rater groups, which implies that certain areas of professionalism can be more easily responsive to targeted sensitization than others.

This paper determined the viability and effectiveness of a 360-degree feedback system in measuring the professionalism of tertiary care hospital residents. We have found this tool relevant and useful to postgraduate medical training. The positive changes in post-test ratings in various areas, especially communication, teamwork, and overall professional conduct, are consistent with the previous research that proposed 360-degree assessments to be useful in evaluating primary ACGME competencies, such as professionalism and interpersonal skills [16-20].

The findings show that a 360-degree tool does not only reveal the professional behaviors; it also acts as a mechanism, which facilitates the process of delivering the feedback in a training setting. This result is in line with other studies that have found the effect of the instrument in faculty confidence and comfort in providing feedback [21]. Moreover, we have demonstrated that the ratings between the various evaluator groups differed and this supports the point that multi-stakeholder feedback is a more holistic evaluation than the traditional faculty-based evaluations [22, 23].

Past studies have reported the significance of applying several assessment modalities including OSCEs, mini-CEX, and simulation to assess various aspects of medical competence with their own advantages and disadvantages [24]. The evaluation methods criteria of validity, reliability, acceptability, and cost-effectiveness and educational impact proposed by Van der Vleuten justify the use of the multi-source feedback as an additional method of evaluation [25].

In our investigation, a tendency of nurses to rank the residents below faculty or peers can be explained by the fact that nurses spend more time and have a closer contact with residents in clinical environments. The same trend has also been found in other research studies, with the ratings of nurses not necessarily in agreement with the faculty ratings [22, 23, 26]. The intra-group difference in nursing personnel can also be attributed to the exposure of nursing staff to different residents in case of an inpatient and outpatient setting [27].

Most of the residents also scored highly on overall scores which may be attributed to either actual high performance or rater bias in a tight-knit training program. The same trends have been observed in the previous studies where the evaluations were situational and in many cases affected by the interpersonal dynamics in the same residency group [28].

Although the rating was mostly positive, the lowest score was registered on the communication between residents and nursing staff (question 1), which is a major point of specific intervention. On the other hand, the maximum score of equitable treatment of patients (question 8) indicates that the residents have a high level of professional responsibility and integrity, which is a good sign of medical practice in the future.

Although the usefulness of the 360-degree evaluations as a formative measure is highly substantiated, its efficacy depends on the formalized strategy, training of raters, and feedback systems [29, 3031]. Research has confirmed that multi-source feedback measures are valid and reliable in measuring professionalism in different specialties when used in a residential setting [32, 33, 34].

On the whole, our results prove that multi-source feedback may provide invaluable information about the behaviour of residents, enhance the feedback culture, and supplement the traditional evaluation tools. The 360-degree can be used to enhance a more detailed and realistic view of the concept of professionalism in clinical practice by including multiple views, in particular, the views of non-physician staff members..

This pilot project is a significant move towards the creation of a culture where faculty members regularly engage in the provision of systematic feedback on professional behaviour and highlights the importance of evaluating and improving professionalism among residents. The continuous assessment of professionalism is believed to promote serious conversation, promote the attitude of residents, and provide the faculty with the opportunity to participate in the feedback process. The 360-degree assessment provides a more in-depth look into resident behavior by adding various viewpoints, which adds depth to the assessment and learning. The information gained in this research can be useful in the overall discussion of the issue of professionalism in medical education and can be utilized in designing future training programs. Such formative assessments are bound to enhance the further maturation of professionalism in a changing healthcare environment as residents continue their professional careers.

1.       American Board of Residential Medicine. Project Professionalism. Philadelphia: ABIM; 1995.

2.       Accreditation Council for Graduate Medical Education (ACGME). Common Program Requirements. ACGME; 2022.

3.       National Medical Commission. Competency Based Postgraduate Curriculum. NMC; 2020.

4.       Stern DT. Measuring medical professionalism. Oxford University Press; 2006.

5.       Medical Professionalism Project. Medical professionalism in the new millennium: a physician charter. Ann Resident Med. 2002;136(3):243–246.

6.       Papadakis MA, et al. Disciplinary action by medical boards and prior behavior in medical school. N Engl J Med. 2005;353(25):2673–2682.

7.       Lockyer J. Multisource feedback in the assessment of physician competencies. J Contin Educ Health Prof. 2003;23(1):4–12.

8.       Ramsey PG, et al. Use of peer ratings to evaluate physician performance. JAMA. 1993;269(13):1655–1660.

9.       Norcini JJ, et al. 360-degree evaluations. In: Holmboe ES, Hawkins RE, editors. Practical guide to the evaluation of clinical competence. Elsevier; 2008.

10.    Violato C, Lockyer J. Self and peer assessment of pediatricians, psychiatrists and medicine specialists: implications for self-directed learning. Adv Health Sci Educ Theory Pract. 2006;11(3):235–244.

11.    Ferguson PC, et al. Development and validation of a 360-degree assessment instrument for residents. Can J Surg. 2003;46(2):105–112.

12.    Cruess RL, Cruess SR, Steinert Y. Teaching Medical Professionalism: Supporting the Development of a Professional Identity. Cambridge University Press; 2016.

13.    Rees C, Knight L, Wilkinson C. Doctors being up there and we are being down here: a metaphorical analysis of talk about student/doctor–patient relationships. Soc Sci Med. 2007;65(4):725–737.

14.    Singh T, et al. Teaching and assessing professionalism in medical learners and practicing doctors. Indian Pediatr. 2016;53(6):495–499.

15.    Adkoli BV, Al-Umran KU, Al-Sheikh M, Deepak KK. Medical students’ perceptions of professionalism: a qualitative study from Saudi Arabia. Med Teach. 2011;33(10):840–845.

16.    Lynch DC, Surdyk PM, Eiser AR. Assessing professionalism: A review of the literature. Med Teach. 2004; 26:366–73.

17.    Musick DW, Mc Dowell SM, Clark N, Salcido R. Pilot study of a 360-degree assessment instrument for physical medicine & rehabilitation residency programs. Am J Phys Med Rehabil. 2003; 82:394–402. [PubMed]

18.    Wood J,Collins J, Burnside ES, et al. Patient, faculty, and self-assessment of radiology resident performance: A 360-degree method of measuring professionalism and interpersonal/communication skills. Acad Radiol. 2004; 11:931–9. [PubMed]

19.    Joshi R, Ling FW, Jaeger J. Assessment of a 360-degree instrument to evaluate residents' competency in interpersonal and communication skills. Acad Med. 2004; 79:458–63. [PubMed]

20.    Higgins RS, Bridges J, Burke JM, O'Donnell MA, Cohen NM, Wilkes SB. Implementing the ACGME general competencies in a cardiothoracic surgery residency program using 360-degree feedback. Ann Thorac Surg. 2004; 77:12–7. [PubMed])

21.    Stark R, Korenstein D, Karani R. Impact of a 360-degree Professionalism Assessment on Faculty Comfort and Skills in Feedback Delivery. J Gen Resident Med. 2008 Jul;23(7):969–72‏.

22.    Ogunyemi D, Gonzalez G, Fong A, Alexander C, Finke D, Donnon T. et al. From the eye of the nurses: 360-degree evaluation of residents. J Contin Educ Health Prof. 2009 Spring;29(2):105–10‏.

23.    Brinkman W. B., Geraghty S. R., Lanphear B. P., et al. Evaluation of resident communication skills and professionalism: a matter of perspective? Pediatrics. 2006;118(4):1371–1379.

24.    Epstein RM. Assessment in Medical Education. N Engl J Med. 2007; 356:387–396‏.

25.    Van Der Vleuten CPM. The assessment of professional competence: developments, research and practical implications. Adv Health Sci Educ. 1996; 1:41–67.

26.    Davis J. D. Comparison of faculty, peer, self, and nurse assessment of obstetrics and gynaecology residents. Obstet Gynecol. 2002;99(4):647–651.

27.    Calhoun J. G., Woolliscroft J. O., ten Haken J. D. Residential medicine house officers' performance as assessed by experts and standardized patients. J Med Educ. 1987;62(9):754–760.

28.    Brainard, A.H., &Bristen, N.C. (2007). Learning Professionalism: A view from the trenches. Acad Med, 82,1010-14

29.    Atkins PW, Wood RE. Self‑Versus others’ ratings as predictors of assessment centre ratings: Validation evidence for 360‑degree feedback programs. Pers Psychol 2006; 55:871‑904.

30.    Overeem K, Wollersheim H, Driessen E, Lombarts K, van de Ven G, Grol R. et al. Doctors’ perceptions of why 360-degree feedback does (not) work: a qualitative study. Med Educ. 2009 Sep;43(9):874–82‏.

31.    Wood L, Hassell A, Whitehouse A, Bullock A, Wall D. A literature review of multi-source feedback systems within and without health services, leading to 10 tips for their successful design? Med Teach. 2006 Nov;28(7): e185–91‏.

32.    Al Khalifa Kh, Ansari AA, Violato C, Donnon T. Multisource Feedback to Assess Surgical Practice: A Systematic Review. J Surg Educ. 2013 Jul-Aug;70(4):475–86‏.

33.    Qu B, Zhao YH, Sun BZ. Assessment of Resident Physicians in Professionalism, Interpersonal and Communication Skills: a Multisource Feedback. Int J Med Sci. 2012;9(3):228–36‏.

34.    Joshi R, Ling FW, Jaeger J. Assessment of a 360-degree instrument to evaluate residents’ competency in interpersonal and communication skills. Acad Med 2004; 79:458‑63.