Mini-clinical evaluation exercise and feedback on postgraduate trainees in the emergency department: A qualitative content analysis

Article Outline

• Abstract
• 1. Introduction
• 2. Methods
  • 2.1. Study design and population
  • 2.2. Study protocol and measures
  • 2.3. Data analyses
• 3. Results
  • 3.1. Medical interview
  • 3.2. Physical examination
  • 3.3. Professionalism
  • 3.4. Clinical judgment
  • 3.5. Counseling skills
  • 3.6. Organization/efficiency
  • 3.7. Clinical procedures
• 4. Discussion
• Acknowledgments
• References
• Copyright

Abstract 

Background/Introduction

Mini-clinical evaluation exercises (mini-CEXs) have been successfully adapted as a formative and summative assessment tool for various different postgraduate medical programs. However, only a few studies have evaluated its use in the setting of an emergency department (ED).

Purpose(s)/Aim(s)

The purpose of this study was to examine the quality of feedback given by preceptors to postgraduate Year 1 (PGY-1) residents during mini-CEXs in EDs.

Methods

This prospective observational study involved EDs at 20 teaching hospitals and comprised 893 mini-CEX encounters involving 230 PGY-1 trainees and 242 preceptors. All feedback forms, which contained three sections, namely “Positive Feedback,” “Negative Feedback,” and “Action Plan,” were assessed using qualitative content analysis techniques.

Results

A total of 734 mini-CEX sessions (82.2%) contained positive feedback, 507 (63.8%) contained negative feedback, 350 (39.2%) contained action plans, and 131 (14.7%) had no feedback. These written feedback comments could be structured into 1,877 coded items and grouped into seven domains of clinical competence. These were: (1) medical interviewing, (2) physical examination, (3) professionalism, (4) clinical judgment, (5) counseling, (6) organization/efficiency, and (7) clinical procedures.

Conclusion

During feedback from the mini-CEXs in the ED setting, preceptors to the PGY-1 students tend to emphasize clinical judgment and seemed to pay less attention to facilitate the development of reflective skills and communication skills.

Keywords: Clinical competence, Feedback, Medical education, Mini-CEX

Back to Article Outline

1. Introduction 

The mini-clinical evaluation exercise (mini-CEX) has been found to be a valid and reliable assessment tool when assessing the core competencies of postgraduate trainees.¹ It was initially developed by the American Board of Internal Medicine (ABIM) and is now globally adopted by other specialties because of it is easy to use, time-efficient, and applicable in many clinical settings.¹, ² Through direct observation, preceptors were able to evaluate the clinical skills of trainees during history taking, physical examination, diagnosis and management. After the clinical encounter, the assessor then has an opportunity to provide evocative real time feedback to trainees. The mini-CEX can be successfully adapted as both a formative and summative assessment tool for postgraduate medical programs.² The feedback process of a formative mini-CEX helps trainees to grow professionally by providing them with insights into their strengths and weaknesses of their skills.³

Medical trainers usually give balanced feedback to trainees according to the Pendleton’s rules. These recommended that, when giving feedback, teachers should first concentrate on positive ideas before giving destructive criticisms.⁴ The learners and teachers should then come up with an action plan in order to achieve their learning objectives. However, it is not uncommon that teachers often miss the feedback session or fail to give organized feedback content.⁵, ⁶ Fernando and colleagues⁵ analyzed feedback given by preceptors in 396 mini-CEX encounters and found that positive features could not be identified in 22.7% of feedbacks, suggestions for improvement were not highlighted in 28.2% of feedbacks, and action plans were not formulated in 49.7% of feedback. Similar findings were obtained by Holmboe and others⁶ in their study of 107 videotaped feedback sessions of mini-CEXs. A timely and constructive feedback from the preceptor is essential for learner improvement.⁷ The quality of feedback is especially important during formative mini-CEXs. Faculty development programs should incorporate courses on how to provide learners with effective feedback. To date, only a few articles have examined the content of mini-CEX feedback sessions. Holmboe and others⁶ suggested that there are six feedback categories, namely medical interviewing, physical examination, counseling, medical knowledge, professionalism, and other areas. However, in their study, a detailed contexture description for each category was absent and only internal medicine residents were evaluated.

In July 2009, the Department of Health of Taiwan mandated medical graduates to learn general medicine by rotating through Internal Medicine, Surgery, Community Medicine and Emergency Medicine (EM) departments during their first year of postgraduate training. During their 1-month rotation through EM, the curriculum included clinical training in trauma and nontrauma settings and the course evaluation involved using the mini-CEX. The purpose of this study was to examine the quality of feedback given by preceptors to postgraduate Year 1 (PGY-1) residents in the emergency department (ED) during the mini-CEX.

Back to Article Outline

2. Methods 

2.1. Study design and population 

This prospective observational cohort study included 893 mini-CEX encounters in 20 teaching hospitals between July 2009 and November 2009, which involved 230 PGY-1 residents and 242 preceptors. EM preceptors who joined the PGY-1 training program had to be board-certified emergency physicians and had to have completed a faculty-training course (lectures and workshop) that focused on the skills of clinical teaching, evaluation, and feedback. The study required verbal consents from all participants in advance and was approved by the Taiwan Joint Institutional Review Board (JIRB). On average, each PGY-1 resident was required to conduct one mini-CEX per week. Before starting a mini-CEX, preceptors, who were also emergency physicians, were responsible for selecting appropriate patients and for obtaining their verbal consent to the evaluation.

2.2. Study protocol and measures 

We adopted a modified version of the standard ABIM mini-CEX procedure.⁸ During each mini-CEX encounter, the PGY-1 resident carried out a focused history taking or physical examination in the ED under the preceptor’s observation. The preceptor then challenged the resident in terms of diagnostic or therapeutic decisions and concluded the session by providing a quantitative rating and qualitative feedback.

Using a nine-point scale, the preceptor quantitatively rated the resident’s performance under seven domains: (1) medical interviewing, (2) physical examination, (3) professionalism, (4) clinical judgment, (5) counseling, (6) organization/efficiency, and (7) clinical procedures. In this modified version of the mini-CEX, we had substituted the original category of “overall competence” with a new category named “clinical procedures” since the definition of “overall competence” was ambiguous and trainees were frequently required to perform simple “clinical procedures” in the ED (e.g., wound suturing, fracture casting, and bedside ultrasonography). After each encounter, preceptors were requested to give a brief (5–10 minutes) qualitative feedback to trainees, both verbally and in written form, in three sections: (1) positive feedback, (2) negative feedback, and (3) action plan.

Each mini-CEX evaluation form consisted of three copies and the same form was used across all sites of the study. Printed guidelines relating to the rating system and feedback were available to each preceptor in order to maintain consistency of scoring and reduce errors in filling the forms. The preceptor and resident each kept a copy, and the third copy was forwarded to the Education Committee of the hospital.

We did not set a sampling size in our study because it was a qualitative analysis and generally there is no limit on sample size in qualitative research.⁹ The process of sampling continued until we found that novel information was no longer being generated by additional samples. Between July 2009 and November 2009, we collected 893 mini-CEX evaluation forms from 20 teaching hospitals and this process involved 230 trainees and 242 preceptors. Table 1 shows the demographic information on the preceptors. The 230 trainees were PGY-1 residents from different medical specialties: 60 from internal medicine, 43 from emergency medicine, 25 from surgery, 16 from family medicine, 14 from obstetrics, 11 from ophthalmology, 10 from anesthesia, and the remainder from other specialties.

Table 1. Demographic characteristics of the mini-CEX preceptors in emergency medicine (n = 242).

2.3. Data analyses 

Feedback content written in the evaluation forms was analyzed qualitatively using the constant comparative method based on grounded theory.¹⁰ Two researchers (Drs. Lin and Chong) independently worked through a line-by-line data coding process and compared any additional data with previously collected data before continuing with the next iteration of data collection. Consensus between the two authors was reached through extensive discussion of the codes and how the codes were developed. Once all relevant codes were identified, they were grouped together into meaningful categories. These categories were then grouped under appropriate domains, which were used to generate a theory. A detailed record was kept on the audit trail concerning the decision rules regarding the grouping of the categories and development of the domains.¹¹ The results of the analyses were examined by two other researchers (Drs. Chiu and Yen) to verify the validity and efficacy of the process. In addition to qualitative description, the frequency of each category and domain was also calculated and compared.

Back to Article Outline

3. Results 

Our study collected 893 mini-CEX sessions; 734 (82.2%) contained positive feedback, 507 (63.8%) contained negative feedback, 350 (39.2%) contained action plans, and 131 (14.7%) did not include any feedback (Table 2). Using qualitative analysis, the feedback content of the 893 mini-CEX sessions was further divided into 1,877 feedback items. These items were coded and grouped according to the seven domains of clinical competence (Table 3): medical interviewing, physical examination, professionalism, clinical judgment, counseling, organization/efficiency, and clinical procedures. The domain that received the most attention from preceptors was “clinical judgment.” The seven domains of clinical competence are described below. The frequency distributions of specific behaviors under each domain are presented in Table 4.

Table 2. Frequency distribution of the mini-CEX sessions (n = 893).

Table 3. Frequency distribution of feedback items (n = 1877) under the seven domains of clinical competence.

Table 4. Frequency distribution of specific behaviors under the seven domains of clinical competence.

3.1. Medical interview 

The medical interview relates to the resident’s ability to take a proper medical history and their communication skills. Based on the patient’s chief complaint, the resident was expected to collect a comprehensive medical history. For example, trainees should not forget to ask for travel and contact history in a patient with fever or ask for coronary artery disease risk factors in a patient with chest pain. A competent trainee should be able to follow a systematic and logical approach to the information gathering. He or she should also focus on other sources of information other than the patient such as family members and care providers. For communication skills, a competent resident should be able to introduce herself/himself, use open-ended questions, use common language, and respond appropriately to the patient’s concerns. Examples of feedback items were: “Beginning with a problem of vertigo, you should consider other systemic diseases and inquire about other important negative findings and previous medication,” or “You should stay with the main issues and use a logical sequence in your questioning, instead of being lead astray by the patient,” or “For a bed-ridden patient, it is essential to collect information from multiple sources, including the caregivers from the nursing home.”

3.2. Physical examination 

For the physical examination (PE), evaluators were observe to determine if they carried out appropriate behavior in relation to patient privacy, correct and comprehensiveness of examination skills, a proper and logical sequence in the examination, and proper hygiene such as hand washing and wearing of a mask. Examples of feedback items were: “It is commendable that you did not forget to close up the curtain before doing the PE,” or “You should make it a habit of explaining to patient what you were doing even to a patient who is unconscious,” or “You should follow the proper procedure for a PE. For example, you should ask a patient to lie down when you do an abdominal examination,” or “In the ED, even though we use focused examinations with symptoms as the main lead, we should not ignore the systematic approach. When assessing a trauma patient, you should focus not only on the injured area but examine the whole body systematically,” or “When you suspect a patient may have a H1N1 infection, you should pay attention to how to protect yourself.”

3.3. Professionalism 

In this domain, preceptors evaluated the following two areas: (1) the doctor–patient relationship where residents should convey empathy to patients, actively seek a solution to patient’s problem, express patience, show respect, and show friendliness even in a busy ED, and (2) interprofessional relationships where residents were expected to interact with nurses, pharmacists, laboratory technicians, and clinicians from different departments or hospitals in a proper manner, and maintain an active learning attitude by seeking help from senior staff or other specialists. Examples of feedback items were: “In a busy ED, you can maintain your good behavior by showing empathy to your patient,” or “Care of elderly patients should be comprehensive. You should not focus only on their main physical complaint,” or “Maintain a confident and pleasant demeanor when you are examining a patient. Do not get nervous,” or “Your interaction with other health professionals should be harmonious,” or “When you are explaining a medical condition to a patient transferred from another hospital, you should avoid any conflict in opinion.”

3.4. Clinical judgment 

The domain of clinical judgment places emphasis on the trainees’ clinical knowledge and their ability to prioritize life-threatening conditions, to make differential diagnoses, to manage patient’s illness and injuries, and to solve clinical problems using an evidence-based medicine approach. Examples of feedback items were: “To improve your knowledge in emergency medicine, in addition to reading textbooks, taking courses in Advanced Trauma Life Support® (ATLS) and Advanced Cardiac Life Support® (ACLS) are most beneficial,” or “When assessing a patient with abdominal pain, it is important to rule out a surgical condition first,” or “You should not diagnose low-pressure headache based purely on history and PE,” or “If you are not sure, search PubMed for more evidence. Self-directed learning is crucial for you to improve your knowledge.”

3.5. Counseling skills 

In the domain of counseling skills, preceptors evaluated the residents for their skills in: (1) patient education regarding management and prevention strategies, and (2) explaining to patients the risks and benefits of a specific treatment. Examples of feedback items were: “If a patient’s condition is expected to deteriorate, you should explain this to family members in advance,” or “You should explain to the patient that taking nitroglycerin (NTG) often carries the side effect of headache,” or “You should reconsider why computed tomography is required and explain to patient the risks associated with radiocontrast medium.”

3.6. Organization/efficiency 

For organization skills, preceptors examined the ability of the residents to make prioritized decisions in a limited time. Residents who are “efficient” should be familiar with the proper workflow of diagnostic tests, imaging, and disposition to avoid stagnation in the patient flow. Examples of feedback items were: “For major trauma, you have to complete the primary survey before starting the secondary survey,” or “When treating myocardial infarction, you should obtain an immediate consultation with the cardiologist in order to shorten the door-to-balloon time,” or “Bring the chart to the bedside so that you do not miss anything during interview,” or “Please be familiar with the hospital’s computer system so that you can manage your patients more quickly.”

3.7. Clinical procedures 

Trainees’ competencies in different clinical procedures can be evaluated during mini-CEX. These include wound suturing, applying a neck collar, inserting a chest tube or endotracheal tube, reduction and fixation of fractured bone, and ultrasonography. For their feedback, in addition to pointing out the strengths and weaknesses of the residents, preceptors also suggested other action plans such as participation in skill workshops (e.g., ultrasonography) to obtain more practice. Examples of feedback items were: “When inserting the endotracheal tube, wait until the vocal cords are open before you pass the tube,” or “You can improve your skills in wound suture by attending the suture workshop,” or “You should obtain a signed informed consent for thoracocentesis from the patient before the procedure.”

Back to Article Outline

4. Discussion 

Providing effective feedback to adult learners in the ED can be challenging. Mini-CEX is both an evaluation and an education tool in medical training. However, there are only a few studies that have focused on preceptors’ skills and the content of feedback.⁵, ⁶ Our study provides valuable information in three areas. Firstly, we have revised the mini-CEX feedback form to include “Positive Feedback,” “Negative Feedback,” and “Action Plan” to remind the preceptors to be constructive and interactive during feedback, which also helped with our data collection. Secondly, our study included a large sample size and involved the EDs of 20 teaching hospitals and comprised 983 mini-CEX encounters involving 230 PGY-1 trainees and 242 preceptors. Finally, the results of our study revealed the behavior of faculty in emergency medicine as they provide feedback as part of a mini-CEX.

The domain of clinical competence that received most attention from emergency medicine preceptors was “clinical judgment,” which received 50.3% of their feedback items grouped into the “action plan” category (Table 4). On the other hand, Holmboe and others⁹ studied the feedback content given to internal medicine residents during mini-CEX in an outpatient setting and found that most of the recommendations were related to medical interview, physical examination, and counseling. Thus, their results are different from ours in two respects. Firstly, different clinical settings (outpatient vs. ED) may affect the clinical performance of the learners differently. The patient approach used in the ED is usually one that is problem-oriented or symptom-oriented; it also requires strong clinical judgment because ED patients are heterogeneous and their definite diagnoses are not apparent. By contrast, patient encounters in an outpatient setting focus less on clinical judgment because the diagnosis of a patient is usually known in advance. Secondly, the learners’ background is also different. Residents who were part of the Holmboe’s study were treating patients only in their own area of expertise (internal medicine).⁶ By contrast, the PGY-1 trainees that formed our study were in the early part of their residency training and were treating patients with unspecified diseases; as a result, they are likely to show greater inadequacy when making clinical judgment.

In this study, preceptors did not provide any written feedback in 131 mini-CEX sessions (14.7%). For those sessions with feedback in print, only 63.8% had documented areas needing improvement and only 39.2% of the sessions had an “action plan.” According to Pendleton’s rule, feedback content should be constructive to facilitate self-reflection from the learners but our results showed that the preceptors tended to give compliments rather than give constructive criticism. This result is similar to that obtained by Holmboe⁶ who found that only 10% of the feedbacks contained an “action plan.” Feedback quality from EM faculty may be influenced by the stressful environment of the ED.¹² Although we had purposely included an “action plan” section in our mini-CEX form to make it easier for preceptors to provide structured feedback, the outcome was disappointing. Our assumption is that there are still many ED preceptors who are not familiar with the content and importance of “action plans” during mini-CEX feedback.⁴ To improve the outcome of medical education, faculty development training in the area of the mini-CEX should place more emphasis on techniques that help preceptors to provide constructive criticism to learners.⁷

The six general competencies of Accreditation Council for Graduate Medical Education¹³, ¹⁴ provide an excellent guide for us to examine the quality of the feedback content of our study. We found that the mini-CEX feedback from ED preceptors seemed to be generally inadequate in two areas, namely “communication skills” and “professionalism.” Specifically, ED preceptors seem to largely ignore area such as: (1) how to listen effectively, (2) how to assess a patient’s understanding of problem and desire for more information, (3) how to make informed decision based on clinical information and patient’s preferences, and (4) how to provide reflective skills. One possibility may be that the preceptors failed to seize the appropriate timing and opportunity to provide feedback in these areas.¹⁵ Noel and colleagues¹⁶ in their study on the traditional complete CEX found that faculty failed to detect 68% of errors committed by residents. Elliot and others¹⁷ found that faculty observers did not reliably evaluate 32% of the physical examination skills among students. These results and our results suggested that a greater emphasis on faculty development in the areas of clinical skills and evaluation is needed.

The need to improve the communication skill of clinical faculty was also demonstrated in the study of Lin and colleagues,¹⁸ which examined the expression of empathy by emergency physicians. Many emergency physicians are not aware of the effects of nonverbal communication on their patients, such as conversations while their eyes look at a computer screen or conversation without asking if the patient understood the problem or the treatment plan. Braddock and colleagues¹⁹ found that among 1,057 counseling sessions involving primary care physicians and surgeons, only 9% of the encounters met the basic criteria for effective informed decision making. After the introduction of National Health Insurance, medical costs are no longer an issue of health access in Taiwan.²⁰ As a result, physicians may overlook the importance of communication with their patients because patients with NHI tend to agree with the treatment plan suggested by physicians.

The development of a self-reflective skill is crucial for residents to grow professionally. However, preceptors often failed to endorse this clinical skill during their feedback.⁴, ⁷ Cruess and colleagues²¹ developed a “professionalism mini-evaluation exercise,” which is similar to mini-CEX, but focused particularly on the evaluation of professionalism. The four skills evaluated were: (1) doctor–patient relationship skills, (2) interprofessional relationship skills, (3) time management, and (4) reflective skills. There is no doubt that evaluation of reflective skills can be and should be a component the mini-CEX.⁴, ²¹

Holmboe⁶ found that, of the 107 mini-CEX sessions, only in 38% of the sessions did the preceptors asked the residents to engage in self-assessment. In our study, no preceptor commented on the reflective skills of the residents. One of the barriers to teaching reflective skills in the ED is time limitation.¹² Even though reflection may require a more in-depth discussion of a problem and therefore requires more time, nevertheless this process can be shortened by focusing on essential points if time is limited. The essence of teaching this skill to faculty is that, firstly, clinical teachers should realize that, from the point of view of the learners, development of reflection skills is an important part of their professional growth.+ Secondly, clinical faculty should learn how to seize the appropriate opportunity to use leading questions such as “What did we accomplish with this interaction?” or “What did we learn from this encounter?” to help the learners to develop their reflection skills.⁴

The mini-CEX uses seven clinical competencies as the bases for the evaluation of the learners, but it is difficult for the preceptors to know what kind of specific behavior to look for under each competency, especially when the environment is different (e.g., outpatient department (OPD) vs. ED). One solution is to convene a panel of experts to generate a list of guidelines as evaluation criteria, but the validity of these guidelines need to be tested using the mini-CEX.²² Our study showed that during medical interviewing, ED preceptors placed more emphasis on the comprehensiveness and skill of questioning, while tending to ignore skills related to listening effectively, how to assess a patient’s needs, and how to make a clinical decision based on the patient’s preferences. Steinert²³ pointed out that for faculty development workshops, it is important to incorporate the special characteristic of the learning setting so that environment specific training can be provided to the faculty. Our study provides useful information to educators and program directors who are involved in the setup of mini-CEX in the ED.

One limitation of our study is that we based our results on what was written on the structured mini-CEX forms. It is possible that there were verbal feedbacks that were not written down on paper. Specifically, the stressful working environment of an ED may be one of the obstacles to ED physicians filling in these forms.¹² Another limitation is that we did not do quality control on the observational skills. This limitation is highlighted by our findings that no preceptors commented on reflection skills and some aspects of communication skills.

In conclusion, our study shows that during mini-CEX feedback in an ED setting, preceptors of PGY-1 residents tend to emphasize clinical judgment and pay less attention to facilitate the development of reflective skills and communication skills by the students. These findings and the specific behaviors we observed under the various clinical competencies should serve as useful resources for medical educators in their design of future mini-CEX.

Back to Article Outline

Acknowledgments 

This study was supported by Taiwan Joint Commission on Hospital Accreditation and Quality Improvement. The authors thank Robert M.K.W. Lee, PhD, McMaster University, Canada, for assistance with manuscript development and the English revision. We also acknowledge Professor Walter Chen of China Medical University (Taiwan), who helped us to revise the mini-CEX form, conducted workshops for us, and offered us much support and encouragement throughout the project.

Back to Article Outline

References 

1. Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA. 2009;302:1316–1326
   • View In Article
   • CrossRef
2. Norcini JJ, Blank LL, Duffy FD, Fortna GS. The mini-CEX: a method for assessing clinical skills. Ann Intern Med. 2003;138:476–481
   • View In Article
3. Malhotra S, Hatala R, Courneya CA. Internal medicine residents’ perceptions of the Mini-Clinical Evaluation Exercise. Med Teach. 2008;30:414–419
   • View In Article
   • CrossRef
4. Branch WT, Paranjape A. Feedback and reflection: teaching methods for clinical settings. Acad Med. 2002;77:1185–1188
   • View In Article
   • MEDLINE
   • CrossRef
5. Fernando N, Cleland J, McKenzie H, Cassar K. Identifying the factors that determine feedback given to undergraduate medical students following formative mini-CEX assessments. Med Educ. 2008;42:89–95
   • View In Article
6. Holmboe ES, Yepes M, Williams F, et al. Feedback and the mini clinical evaluation exercise. J Gen Intern Med. 2004;19:558–561
   • View In Article
   • MEDLINE
   • CrossRef
7. Bing-You RG, Trowbridge RL. Why medical educators may be failing at feedback. JAMA. 2009;302:1330–1331
   • View In Article
   • CrossRef
8. American Board of Internal Medicine. Mini-CEX. http://www.abim.org/program-directors-administrators/assessment-tools/mini-cex.aspx. [accessed 15.08.11].
   • View In Article
9. Tuckett AG. Qualitative research sampling: the very real complexities. Nurse Res. 2004;12:47–61
   • View In Article
   • MEDLINE
10. Pope C, Ziebland S, Mays N. Analyzing qualitative data. BMJ. 2000;320:114–116
    • View In Article
    • MEDLINE
    • CrossRef
11. Mays N, Pope C. Assessing quality in qualitative research. BMJ. 2000;320:50–52
    • View In Article
    • MEDLINE
    • CrossRef
12. Yarris LM, Linden JA, Gene Hern H, et al. Attending and resident satisfaction with feedback in the ED. Acad Emerg Med. 2009;16(Suppl. 2):S76–S81
    • View In Article
    • CrossRef
13. Swing SR. The ACGME outcome project: retrospective and prospective. Med Teach. 2007;29(7):648–654
    • View In Article
    • CrossRef
14. Duffy FD, Gordon GH, Whelan G, et al. Assessing competence in communication and interpersonal skills: the Kalamazoo II report. Acad Med. 2004;79:495–507
    • View In Article
    • MEDLINE
    • CrossRef
15. Holmboe ES. Faculty and the observation of trainees’ clinical Skills: problems and opportunities. Acad Med. 2009;84:551–558
    • View In Article
    • CrossRef
16. Noel GL, Herbers JE, Caplow MP, Cooper GS, Pangaro LN, Harvey J. How well do internal medicine faculty members evaluate the clinical skills of residents?. Ann Intern Med. 1992;117:757–765
    • View In Article
    • MEDLINE
17. Elliot DL, Hickam DH. Evaluation of physical examination skills. Reliability of faculty observers and patient instructors. JAMA. 1987;258:3405–3408
    • View In Article
    • MEDLINE
18. Lin CS, Shu M-YF, Chong CF. Differences between emergency patients and their doctors in the perception of physician’s empathy: implications for medical education. Educ Health. 2008;21:144
    • View In Article
19. Braddock CH, Edwards KA, Hasenberg NM, Laidley TL, Levinson W. Informed decision making in outpatient practice: time to get back to basics. JAMA. 1999;282:2313–2320
    • View In Article
    • MEDLINE
    • CrossRef
20. Chen L, Yip W, Chang MC, et al. The effects of Taiwan’s National health insurance on access and health status of the elderly. Health Econ. 2007;16:223–242
    • View In Article
    • MEDLINE
    • CrossRef
21. Cruess R, McIlroy JH, Cruess S, Ginsburg S, Steinert Y. The professionalism mini-evaluation exercise: a preliminary investigation. Acad Med. 2006;81:S74–S78
    • View In Article
    • MEDLINE
    • CrossRef
22. Margolis MJ, Clauser BE, Cuddy MM, et al. Use of the mini-clinical evaluation exercise to rate examinee performance on a multiple-station clinical skills Examination: a validity study. Acad Med. 2006;81:S56–S60
    • View In Article
    • MEDLINE
    • CrossRef
23. Steinert Y. Faculty development: from workshops to communities of practice. Med Teach. 2010;32:425–428
    • View In Article
    • CrossRef