Commentary
Rethinking the Way We Evaluate Trainees
By L. James Nixon, M.D., and Bradley J. Benson, M.D.
Portfolios afford residents and medical students an opportunity to show what they know and can do.
“We learn by doing and realizing what we did.”
—John Dewey
“Hello, Dr. Smart, I’m glad you are interested in joining our practice. I’m the medical director for the clinic. I’ve reviewed your CV and letters of recommendation and am impressed with your credentials. I do have one question, however. Our health system has several major initiatives in progress that are really changing the way we deliver care. We’re rolling out a new EMR, starting a rapid-cycle improvement project to lower the average Hgb A1c of our diabetic population, and initiating 360-degree evaluations of all clinic staff and providers to improve teamwork. How has your medical school and residency training prepared you for this kind of work environment?”
If either of us had had to answer that question just eight years ago when we finished our residencies, we would not have gotten the job. Clearly, the skill set needed to practice medicine has changed during the last 10 years, as have the expectations for graduating practitioners. The impetus for many of these changes was the 2001 Institute of Medicine (IOM) report “Crossing the Quality Chasm,” which revolutionized thinking about patient safety in hospitals and alerted physicians to the importance of standardization of processes, systems thinking, and teamwork in practice. It also prompted medical educators to redefine their goals for educating 21st century health care providers. Educators began to realize that if graduates were to provide care that is safe, timely, effective, equitable, efficient, and patient-centered, as the IOM report called for, medical education needed
reforming.
The Accreditation Council for Graduate Medical Education (ACGME) has set the reform agenda for resident training programs. In 1999, it defined a set of six core areas in which graduating residents need to demonstrate competence: medical knowledge, patient care, interpersonal and professional communication, professionalism, practice-based learning and improvement, and systems-based practice. The last two, in particular, reflect changes in what we expect from physicians. We no longer expect that any individual can know everything about their field. We do, however, expect that they know where to get the latest information and how to apply it to their practice. We also expect them to be able to implement systems that lead to consistent, high-quality patient care. They need to demonstrate that they have the skills required for continuous quality improvement, better chronic disease management, and navigation of an increasingly complex health care delivery system.
Educators are now looking for new ways to teach these skills. And they are looking for ways to measure competency so that they know whether their programs produce the intended outcomes.
In other words, medical educators across the country are recognizing the need for better ways to develop and measure the characteristics of “the good doctor.”
Assessing Competence
Assessing competence can be done in a number of ways, including exams, 360-degree evaluations in which co-workers and peers rate a resident on communication skills, and chart reviews. These assessments provide trainees with valuable feedback about their knowledge and skills that can aid in their development. But tools for learning need a toolbox. And this is where the portfolio comes in.
The University of Minnesota Medical School is pioneering the development of an electronic performance portfolio that students will be able to use throughout their medical education and ultimately for continuing professional development. The project is part of Medical Education Development 2010—or MED 2010—which is the University of Minnesota’s educational reform initiative. Under the direction of Dean Deborah Powell, M.D., it will fundamentally change the way we educate physicians in the 21st century. The University of Minnesota is also working with the ACGME to help develop a portfolio that could be used by all medical schools and residency programs for performance assessment and accreditation. The idea is for students to use the portfolio during medical school and as they move on to residency, then fellowship, then practice.
What is a Portfolio?
The term portfolio comes from the Latin portare, which mean “to carry,” and folium, which means “sheet or leaf.” Portfolio literally means “to carry sheets.” This is why when we hear the term we think of the artist or architect carrying around a large folder of his or her work.
A portfolio serves two purposes in the art world. First, the artist can learn from his or her portfolio. Imagine the artist pulling out her paintings and reviewing work done 10 years ago during her blue period or reviewing the still life she was doing last year in order to evaluate how her skills have evolved. Second, portfolios are a way for an individual to present his or her work to others. Imagine an artist opening her portfolio to show the types and quality of the work she has done.
But how can portfolios be used in medical education? Ideally, they make it easy for students and residents to store and retrieve their academic records and documentation of accomplishments, records of procedures done, lectures given, and posters presented. It is way for young physicians to document improvements in their clinical practice, for example, how treatment of a cohort of diabetic patients caused their A1c to decrease from 8.4 percent on average to 6.8 percent after two years. A portfolio is also a place where a resident or student can place evidence of their history-taking and medical decision-making abilities. The portfolio thus provides a more complete view of what it takes to be a competent physician than multiple-choice tests or other traditional assessment vehicles.
How are Portfolios Used?
Portfolios can be paper or electronic. Students on the medicine clerkship at the University of Minnesota have a pocket-sized binder that serves as their portfolio. They are given the portfolio at the beginning of the clerkship and turn it in completed at the end. Students set goals such as improving their physical exam skills, medical knowledge, or medical decision-making. These goals are documented in the portfolio. The portfolio also includes documents that show how they asked and answered questions about their patients, how they performed structured clinical observations, mid-rotation and end-of-rotation self-evaluations, and free-form reflections about their experiences with patients or on the wards.
The University of Minnesota med-peds residency program uses an electronic portfolio. Information is accessible to the resident and program director over the Internet. This gives residents the ability to generate and store a large amount of data over a long period. The University of Minnesota has agreed to let residents use these portfolios to document their experiences over the course of their careers.
In the med-peds program, the program director schedules portfolio reviews to give residents the opportunity to see where they have been and where they are going, and to make course corrections to ensure they reach their desired career destination. Later, when applying for jobs or fellowships, they can use the portfolio to illustrate their experience and competency in areas of medicine.
So when Dr. Smart is asked how his medical education prepared him for the work that lies ahead, he’ll be able to refer to his portfolio and answer with confidence.
“Thank you for the opportunity. If I can just log on to your computer, I can show you. You mentioned the EMR rollout. I’ve completed training on EPIC, Allscripts, and CPRS. Here are a few examples of my notes using each of those systems. As for the rapid-cycle diabetes QI project, here is the progress I made with a panel of diabetics during my four years in clinic. You can see it took a while, but the average A1c made it below 7 percent. Last, I look forward to 360-degree evaluations. Although the first one I was involved with was a bit uncomfortable, I learned some valuable lessons about how I come across to my colleagues. It really helped me in the long run. You can see that my final evaluations from patients, nurses, and peers demonstrate that I’m a team player. Yours sounds like exactly the kind of practice environment I’m looking for.” MM
James Nixon and Bradley Benson are assistant professors in the departments of internal medicine and pediatrics at the University of Minnesota Medical School. Nixon directs the internal medicine and pediatric clerkships, and Benson directs the med-peds residency program and is vice chair of the medical school’s Graduate Medical Education Committee.