2014 Spring: Innovations in Assessment

Innovations in health care education require the simultaneous development of creative new approaches to student assessment. This IAMSE webcast seminar series will explore these issues and provide several examples of how schools are evolving with assessment approaches. Many, if not most, contemporary health science curricula are exploring the integrated learning of multiple basic science disciplines in the clinical and social contexts of patient care. These teaching and learning approaches require new and creative assessment tools to monitor student progress. Faculty training to develop and implement these changes is also critical since constructing high quality, reliable assessment tools is a learned skill. Assessment includes formative feedback to students as an effective learning activity, and this is increasingly incorporated into modern curricula. Students are also being taught life-long learning skills and are expected to be independent learners. Yet how can faculty or accreditation agencies be assured of progress towards continued professional growth and competency? As post-curricular, graduate medical educational programs change to more benchmark approaches to measure competency, how will this impact the approaches to learning and assessment in medical and other health science curricula? These and other issues will be the focus of our five seminars. The first session will update us on use of NBME customized exams for basic science assessment, introduce the GEMS program that allows customization for local use, and discuss the COMAT exam for clinical clerkships. Creative pilot programs such as using pharmaceutical ads for interpretations skills will also be discussed. The next three seminars will explore the practical application of evaluation approaches such as the value of exam reviews, formative quizzes, strategies and methods to assess individual learning, and creation and use of student portfolios. Our fifth session will conclude the series with a backward look and discussion of the potential impact of graduate medical educational assessment of competencies with programs like “Milestones” on the development of future undergraduate medical curricula. Its potential impact may be compared to the importance that the ACGME core competencies had on developing curricular objectives.

Back to Archives

February 27, 2014 at 12:00 pm

Newer Approaches to Medical Student Assessments

Presenter: Bruce P. Bates and Melanie V. Nelson

Bruce P. BatesDr. Bates is the Senior Vice President for the National Board of Osteopathic Medical Examiners, responsible for the oversight of the organization’s cognitive examinations which includes the self-assessment COMSAE, the end-of-service subject exams COMAT, and the COMLEX-USA three-level licensure series for osteopathic medicine.

Dr. Bates’ career spans more than 35 years of clinical practice and medical education including residency director, clinical medical direction, and academic medicine.  He joined the NBOME from the University of New England College of Osteopathic Medicine where he had been the Associate Dean for Clinical Services and Chair of the Department of Geriatric Medicine.  He was the president of the medical staff for Southern Maine Medical Center.  Certified in Family Medicine, Geriatric Medicine, and Hospice and Palliative Care, he has also served as Interim Dean and Associate Dean of Clinical Affairs, as well as Medical Director for both SolAmor Hospice of Maine and MatureCare, a multi-professional group dedicated to long term care.


Melanie V. Nelson

Melanie has been with the National Board of Medical Examiners® since 1970. Over the years she has worn many hats, starting as a secretary in the Psychometrics unit and gradually moving up to other roles, which have included operational responsibility for the NBME Subject Examination Program, management of the NBME Services Portal and, most recently, directing the transition of the paper subject examinations to web-based test delivery.  She has also played an active role in NBME’s liaison efforts with medical schools and national student groups over the years, delivering presentations about USMLE and NBME programs and staffing exhibit booths.  Melanie now spends her time managing the NBME Customized Assessment Services (CAS) and Global Evaluation Management System (GEMS), two online platforms for the design, construction and delivery of local web-based examinations.

Bruce Bates, D.O. Senior Vice President for Cognitive Testing of the National Board of Osteopathic Medical Examiners (NBOME) will present on the mission and assessment products of the NBOME with a description of one new product and a new product under development. The Comprehensive Osteopathic Medical Achievement Test (COMAT), an end of service assessment utilization, acceptance and performance will be described along with a brief outline of its correlation with licensing and certification performance. Modeled on a process utilized by The Medical Council of Canada, a new product, Clinical Decision Making (CDM, that is under development will apply a key features approach in an open ended and short answer format for clinical presentations scenarios. The concept development and pilot process will be discussed.

The National Board of Medical Examiners® (NBME®) offers two online platforms for the design, construction and delivery of local web-based examinations: Customized Assessment Services (CAS) and the Global Evaluation Management System™ (GEMS™).  Melanie Nelson, Program Manager for Products & Services, will offer participants a look into how CAS, which offers a pool of 10,000+ high quality, secure items assembled from the NBME basic science bank, has helped faculty develop exams to meet a variety of assessment needs.   Participants will also be provided with an example of how one school has used the GEMS versatile end-to-end assessment management to link items in a Nervous Systems course in a way that allows students and faculty to evaluate study habits and use resources to improve specific learning styles.

If you have any additional questions after participating in this session, please email BBater@nbome.org

Seminar Archive
March 6, 2014 at 12:00 pm

Individual Learning Assessment Approaches

Presenter: Daniel M. Clinchot

Daniel M. ClinchotDr. Clinchot serves as the Vice Dean for Education at the Ohio State University College of Medicine.  He is also an Associate Vice President for Health Sciences at the OSU Wexner Medical Center.  Dr. Clinchot is the architect of the new Lead.Serve.Inspire UGM curriculum at Ohio State.  He is an associate professor with tenure in the Department of Physical Medicine and Rehabilitation.  He earned his MD degree at the Health Sciences Center at Syracuse of the State University of New York and served an internship in internal medicine and a residency in physical medicine and rehabilitation at Ohio State University Medical Center. He has served as the associate residency program director for PM&R from 1998-2001, and as program director from 2001-2009.  Dr. Clinchot has been named teacher of the year by PM&R residents multiple times and in June 2006, Dr. Clinchot was presented with the Excellence in Teaching Award for outstanding teaching in the College of Medicine.  In addition he has received the College of Medicine Alumni Excellence in Teaching Award.

Dr. Clinchot directed the chronic care curriculum for medical students from1994-2003. He also has directed several medical education programs at the College of Medicine, including the College’s Medical Humanities and Behavioral Science program; the Physician Development program and the Medical Ethics, Practice of Medicine program. Dr. Clinchot has been honored as part of the Interdisciplinary Team Award for Excellence in Patient Education.  Within the College of Medicine Dr. Clinchot has served as the Associate Dean for Clinical Education and Outreach and Medical Director of the Clinical Skills Education and Assessment Center from 2003-2006 and then as the Associate Dean for Medical Education from 2006-2012.  A dedicated researcher, Dr. Clinchot has authored over 30 scientific publications and three book chapters.  He frequently lectures at the local, regional, national and international venues.

The recent trend in competency based undergraduate medical education provides the opportunity for individualized teaching and learning and the development of innovatively integrated curricula.  When this framework is coupled with dynamic competency based multimodal learner assessments it becomes structurally sound and transferable.  Accomplishing this however, is not an easy task.  Ensuring that learner assessments of knowledge, skills, attitudes and behaviors are valid and faculty are reliable in their assessments requires diligence and planning.

As undergraduate medical education curricula change assessment is often thought of in isolation.  This commonly results in mal-alignment in the curriculum and dissonance among learners and educators alike.  Creating an assessment framework alongside competency and curriculum development allows for a product that is seen as seamless and integrated by the learner.  Creating a structure that supports individualized assessment can seem daunting, however it does not need to be.  Practical step-wise development of an individualized assessment paradigm over time allows for both faculty and students to adjust and mature alongside the educational program.  Clear connections between competencies, curricular content and assessment establishes confidence for both faculty and learners as well as those who receive our learners in the next phase of their education.

Seminar Archive
March 13, 2014 at 12:00 pm

Pivio: A New Records Management and Life-Long Learning Tool for the Medical Community

Presenter: Mary Drescher Halicki and Angelique Johnson

Mary (Drescher) Halicki is the Relationship Manager for the Association of American Medical Colleges (AAMC) on the Pivio team, responsible for establishing and maintaining relationships with key internal and external stakeholders including medical schools and applicant organizations as well as managing the marketing activities for the Pivio program.
Prior to her time at the AAMC, Mary worked at SmartBrief, Inc., an email newsletter service organization for association and non-profit communications, as the Director of Association Relations where she served as the primary point-person for all healthcare and education association client relationships. She spearheaded efforts to grow relationships within associations and increase overall product utilization, client engagement, subscriber circulation and communication channels.

Prior to her work at SmartBrief, Mary served as an Account Manager for the Burgess Group, a Medicare reimbursement software company, overseeing and managing the company’s client relationships and led technology training and all account management efforts to increase engagement and product utilization. Prior to that, Mary worked at The Advisory Board Company, a global healthcare and education best-practice research and technology firm, in marketing and member services roles, launching the first Account Management function for the Higher Education membership team. In addition, Mary led efforts to foster relationships with various healthcare organizations both through marketing and client services roles.

Mary graduated cum laude with honors in Psychology from Colgate University. She currently lives in Arlington, VA with her husband Kevin.


 

Angelique Johnson is the Director of Operations for Pivio. In this role she participates in the day-to-day management of Pivio policies and procedures and actively engages in the operational issues for this new and exciting service. Angelique has been with the Association of American Medical Colleges (AAMC) since 2006. She began her career at the AAMC with the Electronic Residency Application Service (ERAS) serving as the Manager, Communications and Administrative Services. In that role, she provided internal support to cross-functional AAMC teams, in addition to serving as primary point of contact to staff at the NBME, NBOME, AOA, and ECFMG. After several years in that role, she transitioned into a new position within ERAS – Manager, Training Programs and Business Partner Relations. In this exciting role, she managed the relationship team who served as primary point of contact to over 5000 training programs and ERAS business partners including the NBME and NBOME. Keeping in a trend of service and mission-driven work, at the end of 2012 she moved into her new role with the Pivio team. She enjoys the collaborative and collegial environment at the AAMC and is enthusiastic about the potential for Pivio. She can be contacted at amjohnson@aamc.org.

An overview of AAMC and NBME’s efforts to create a lifelong learning platform that stores and transfers data necessary for pre-medical and medical students, residents and physicians across their careers called Pivio™ (formerly eFolio connector). The project currently contains a platform for users to share and store data at an individual’s control (CV/resumes, exam applications and results, biographical information, etc.). The session will include a demonstration of current features, overview of results to date from the pilot users, and open feedback on future program considerations to meet the needs of medical education.

Seminar Archive
March 20, 2014 at 12:00 pm

Assessment Review: Learning From Mistakes

Presenter: Bobbe Baggio

Bobbe BaggioBobbe Baggio is a consultant in multicultural and international e-Learning integration for a global and virtually connected workforce. Her area of expertise is the integration of technologies to enhance human performance including adult and workplace learning.

Dr. Baggio is particularly interested in helping organizations:

  • Create a wide variety of training and learning solutions including VLEs, VILT’s, distance learning, remote classrooms, multimedia e-learning, virtual simulations, m-learning and compliance training programs, products and courses.
  • Create strategies for the effective use of technologies to enhance learning and human
  • Implement leadership skills and manage a complex multi-tiered virtual environment.
  • Foster virtual relationships across all areas of complex organizations through the development and adherence to organizational objectives and measurable behavior skills.
  • Support both virtual workers and those responsible for managing and leading in the virtual workplace
  • She draws upon her experience as a Fortune 100 IT manager, 20 years of consulting experience, and her doctoral studies in instructional design for online learning.

Current clients include the University of Pennsylvania, PHEAA, Merck, BMS, KPMG, Siemens, Ticketmaster, Christiana Care Health System, Cisco and Adobe

Bobbe Baggio is Director of the Masters of Science program in Instructional Technology Management at La Salle University in Philadelphia, PA.  Since 2002 she has been CEO of Advantage Learning Technologies, Inc. a company that provides programs, products and research for workplace learning in the 21st century. She believes that technologies are here to help everyone and to be used to enhance out human performance. Bobbe is a highly regarded author, speaker, advisor and educator in the field of instructional technologies and learning.

 

Assessment review gives both the instructor and the learners an opportunity to learn from mistakes. By letting your learners review tests, quizzes and even final exams, we are assessing our assessments. This gives you an opportunity to go over the goals of the instruction, assess learner progress and discover information to improve both the assessment and the instruction. To answer the question: “How are they doing?” we need to step back and evaluate the entire instructional effort. In this presentation participants will review the benefits of using all assessments as an opportunity to learn. We will look at each and every assessment as an opportunity. We will discuss strategies for planning and conducting assessment reviews and adapting instructional strategies to enhance learning results. We will explore hands on examples of test review and instructional adjustments. This presentation will allow the participants to see the value of assessment as a tool for learning and provide hands on examples of the feedback loop for improving instruction.

Learning Objectives:

  • Identify strengths of different types of assessments
  • Demonstrate the use of assessment audits
  • Gain strategies for using assessment to improve overall results
  • Consider ways to improve student learning and instruction using assessment

March 27, 2014 at 12:00 am

ACGME “Milestones”

Presenter: William Iobst

William IobstWilliam F. Iobst, MD, a board certified rheumatologist, is currently the Vice President for Academic and Clinical Affairs and Vice Dean at the Commonwealth Medical College. Prior to this position, he served as Vice President of Academic Affairs for the American Board of Internal Medicine. In that capacity, Dr. Iobst developed outreach activities with residency and fellowship training programs to enhance the standard of assessment of competence. He also oversaw the credentialing of ABIM diplomates. In addition to serving as the ABIM ex-officio representative to the ACGME IM RRC, he worked closely with the internal medicine education community to develop competency-based milestones and frameworks for evaluating competency having chaired the ABIM/ACGME Internal Medicine Milestones Writing Group. He also oversaw the development of the Internal Medicine Subspecialty Milestones.

Previously, Dr. Iobst served as Designated Institutional Official and Medical Director of Graduate Medical Education at Lehigh Valley Hospital Network in Allentown, PA. He was also the Vice Chairman of Education for the Department of Medicine, and served as the Medical Director for Network Faculty Development and as an Associate Program Director for the Internal Medicine Residency. Prior to assuming these responsibilities, he served Lehigh Valley Hospital as the Program Director for the Internal Medicine and Transitional Residency Programs and as an Associate Professor of Clinical Medicine at The Pennsylvania State University College of Medicine in Hershey, PA.

Dr. Iobst is a graduate of Bucknell University and Jefferson Medical College of Thomas Jefferson University. He completed his residency at Robert Packer Hospital, and his rheumatology fellowship at Dartmouth-Hitchcock Medical Center.

Competency-based medical education (CBME) has been defined as is an outcomes-based approach to the design, implementation, assessment and evaluation of a medical education program using an organizing framework of competencies. One competency framework frequently utilized for this purpose is the ACGME Six General Competencies. The outcome has evolved to include meeting the needs of our highly complex and evolving health care system. One example of this outcome would be to prepare students to meet the goals of the Institute of Medicine’s Triple Aim. To ensure that students are progressing towards appropriate levels of competence, the graduate medical education community has created sets of developmental milestones that define the knowledge, skill and attitudes that define the six general competencies. This community has also embraced the concept of entrustment (Entrustable Professional Activities) as a strategy to inform the work-based assessment of trainees.

This presentation will review basic definitions of CBME, will discuss how milestones and entrustment are being used in graduate medical education and will introduce how these strategies might apply to undergraduate medical education.

Seminar Archive