Executive Summary

Introduction

The National Conference of Bar Examiners’ (NCBE’s) Testing Task Force (TTF) is undertaking a comprehensive, future-focused study to ensure that the bar examination continues to test the knowledge, skills, and abilities required for competent entry-level legal practice in a changing legal profession. The collaborative study involves input from stakeholders at multiple phases and considers the content, format, timing, and delivery method for NCBE’s current tests, which make up all or part of the bar examination in most U.S. jurisdictions: the Multistate Bar Examination (MBE), the Multistate Essay Examination (MEE), and the Multistate Performance Test (MPT). The study also includes the Multistate Professional Responsibility Examination (MPRE), which is administered by NCBE and required for admission in most U.S. jurisdictions.

This Executive Summary provides a synthesis of the stakeholder listening sessions conducted during Phase 1 of the study between November 2018 and June 2019. The purpose of these sessions was to solicit input from various stakeholder groups about characteristics and considerations for the next generation of the bar examination. The sessions were facilitated by ACS Ventures LLC (ACS), an independent psychometric consulting firm. To read a summary of each of the ten listening sessions, download the full report.

Participants

Most of the listening sessions were conducted in-person in conjunction with scheduled meetings or conferences of stakeholder groups. In addition, virtual sessions using Zoom, an online meeting platform, were held for two stakeholder groups. Multiple sessions were offered in each case to maximize opportunities for stakeholders to provide input. Comments were also received by email.

The stakeholders represented included bar administrators, bar examiners, and justices; law school deans and doctrinal, clinical, and academic support faculty; and lawyer members of the American Bar Association (ABA). The meetings or conferences at which in-person sessions were conducted and the virtual sessions are listed in the following table.

Event/Stakeholder Group	Location	Date
Council of Bar Admission Administrators Fall Meeting	Denver, CO	November 2018
Association of American Law Schools Annual Meeting	New Orleans, LA	January 2019
UBE Jurisdictions Forum	San Antonio, TX	January 2019
ABA Midyear Meeting	Las Vegas, NV	January 2019
LSAC/NCBE Legal Educators Conference	Albuquerque, NM	February 2019
NCBE Annual Bar Admissions Conference	San Francisco, CA	May 2019
ABA Diversity and Inclusion Center and Pipeline Council	Virtual	May 2019
Association of Academic Support Educators Conference	Seattle, WA	May 2019
ABA Deans Workshop	Denver, CO	June 2019
ABA Young Lawyers Division	Virtual	June 2019

Method

The purpose of the listening sessions was to listen to stakeholders’ concerns, thoughts, and ideas related to the bar exam of the future. Members of the TTF or NCBE staff gave a welcome and introduction describing the TTF’s research plan and then stayed in the sessions as observers. To avoid influencing or affecting what stakeholders said, the NCBE/TTF observers did not share their opinions or offer corrections during the sessions when, in a few instances, stakeholder comments reflected misperceptions or misinformation about the current bar exam. Such comments may be included in the individual summaries of the sessions, which are appended to this Executive Summary. These comments were appreciated as they served to reinforce the need for NCBE to strengthen communications with stakeholders and continue working to increase assessment literacy.

Following the welcome and introduction, ACS facilitators explained that the participants would be asked to provide input on the content, format, timing, and delivery method of the bar exam and the MPRE. Participants were invited to candidly provide their opinions and were informed that no comments would be attributable to specific participants in any written reports or materials.

ACS provided participants with the context for the research study by explaining that the purpose of the bar examination is to inform licensure decisions. Specifically, that the bar exam is designed and intended to measure the knowledge, skills, and abilities that newly licensed lawyers (NLLs) need for competent entry-level practice.

The questions listed below served as the guiding framework for the listening sessions.

• What aspects of the current bar exam and MPRE do you think should be kept? Why?
• What aspects of the current bar exam and MPRE do you think should be dropped or modified? Why?
• What do you think the next generation of the bar exam and MPRE should be?
• What cautions do you want to share regarding any potential changes to the bar exam and MPRE?
• What else would you like to discuss about the bar exam and MPRE?

Because each listening session included different stakeholders, the discussions reflected the interests of each respective group. This means, for example, that some groups focused more on content while others focused more on format or timing. The diversity of stakeholders and perspectives provided ample opportunities for rich discussion about each of the major topics.

Key Points

In this section we highlight some of the key points that emerged from participants’ input across all listening sessions. It is important to note that the purpose of the Phase 1 activities was not to draw conclusions about what the bar examination of the future should or will be. Rather, the sessions were intended to serve as a starting point by identifying possible options for redesign and to gather input on where greater emphasis might be placed. With this in mind, the key points set out in this Executive Summary should not be viewed as signaling any decisions regarding future changes, but simply as the main ideas voiced with some degree of frequency by stakeholders; the appended session summaries provide greater detail and additional comments that emerged from each. The TTF will continue to consult with stakeholders during Phase 3 to build upon the valuable input received during Phase 1.

Very few, if any, opinions were universally shared by stakeholders. In fact, some suggestions for change were potentially in conflict with one another (e.g., add simulations or mock client interviews and do not increase costs of exams). Additionally, while the intended focus of the sessions was on changes that could be made, in most sessions there were comments supporting various aspects of the current exam program/model, but with suggested opportunities for continued evolution and improvement.

Content

• The MPT is a strength of the current bar exam.
  • The MPT was widely viewed as the component that is most representative of the skills needed for NLLs at the point of entry to practice.
• The MBE generally represents core content NLLs need to know but tests too deeply within the content.
  • The subject areas measured on the MBE were generally viewed as representative of subjects that would be applicable to all NLLs. However, the target level for items on the MBE was viewed by many as going beyond the point of entry-level competency by testing nuanced issues and “exceptions to exceptions to rules.”
• Lawyering skills should be emphasized over subject matter knowledge.
  • Content that focuses on skills such as issue spotting, critical thinking, legal analysis, written and oral communication, and reasoning was considered more applicable to all NLLs. In contrast, content that focuses on subject matter knowledge was viewed by some as requiring memorization of legal rules that lawyers can look up in practice.
• Consider adding legal research skills by providing access to an electronic database of legal resources.
• There are too many subjects covered on the bar examination when considering the MBE and the MEE in combination, and especially when considering the additional subjects tested in some jurisdictions that draft their own essay questions.

Format

• Constructed-response items are preferred over multiple-choice items.
  • The constructed-response format of the MEE and the MPT was viewed as more representative of what NLLs do in practice (i.e., written analysis of legal and factual issues) than the multiple-choice format of the MBE.
  • The MPRE content could be assessed using essays or MPT-like questions as opposed to, or in addition to, the current multiple-choice format.
  • Multiple-choice items are not reflective of the way law is practiced.
  • While multiple-choice items were not widely supported, many stakeholders recognized the benefits that the MBE contributes: objective scoring, reliability of scores, and scaled scores that have consistent meaning over time and across jurisdictions because the exam is equated.
• Consider adding formats such as multimedia videos or mock clients/simulations to provide additional, and more realistic, assessment of important skills needed by NLLs.
  • While using simulations was suggested by stakeholders, the associated downsides of greater subjectivity in grading, the potential for bias, and increased costs were also noted.

Timing

• Consider a “step testing” approach for the bar examination.
  • The MBE could be the first step (test of legal knowledge) and could be completed while candidates are still in law school, when it would be more proximate to when they take core foundational courses.
  • The second step (test of legal skills), taken upon graduation, could be an enhanced MPT that is content-neutral and/or an essay exam that covers a limited range of content or for which legal resources are provided.
  • While the notion of step testing was frequently mentioned by stakeholders, the downsides of step testing were also discussed, such as taking time away from summer employment opportunities or semester coursework and potentially requiring candidates to incur associated costs of hotel, travel, and prep courses for each step. Further, some stakeholders commented that a step exam after the first year of law school would be too early in students’ training because many students would still be processing what they have learned.
• Consider more frequent administrations of the bar examination.
  • Additional administrations of the bar examination could permit candidates to sit for the exam when they are ready, permit failing candidates to retake the exam sooner, and reduce the time to employment after graduation, which would help graduates with student debt.
  • Reducing the time it takes to grade the constructed-response components (essays and performance tests) could also allow passing candidates to begin practicing sooner.
  • It was also acknowledged that more frequent administrations of the exam could require jurisdictions to need more staff and other resources, which could increase costs.

Delivery and Administration

• Mixed feedback on how the exam should be administered/delivered.
  • There was varied support for paper-based testing, computer-based testing, or some combination of these delivery modes. However, it was mentioned that the delivery of the exam should align with law school, training, and practice environments.

Other Comments/Topics

• Support for the increased uniformity and score portability of the UBE.
  • The consistency in subjects tested and the portability of scores are positive features of the Uniform Bar Exam (UBE) and should be maintained. [The UBE is composed of the MEE, MPT,and MBE and results in a score that can be transferred to other UBE jurisdictions.]
  • Increased consistency in grading of the MEE and MPT across UBE jurisdictions could be accomplished through different activities ranging from increased guidance by NCBE on grading practices to centralized grading for constructed-response/essay questions.
• A common passing score across jurisdictions is supported, but not universally.
  • There was support for greater consistency in passing score requirements to communicate a common standard for entry-level competency, particularly for the UBE, but support for maintaining each jurisdiction’s autonomy in setting its passing score was also voiced.
• Consider conjunctive passing score decision rules.
  • When discussing step testing, support was expressed for separate passing scores for different components, permitting the exam to be administered as stand-alone parts or modules that would contribute to an overall pass/fail decision. This approach would be analogous to examinations for licensure used for Certified Public Accountants (CPA Examination) or physicians (United States Medical Licensure Examination [USMLE]).
• Evaluate the impact of change on historically underrepresented populations of examinees.
  • Ensuring that the bar examination is free from racial/ethnic/gender bias is a priority.
• Consider development of a diagnostic or readiness test for delivery earlier in law school.
  • Although not intended to be part of the bar examination, this test would provide information for law school students after the first year to help evaluate their readiness to continue. With increased costs of law school and associated student debt, this approach could provide information for students and schools to help decide whether students should continue working toward a J.D. and, if so, the areas in which they might benefit from additional instruction to prepare for the bar exam.
    Avoid increased costs for candidates to take the exam and increased costs for jurisdictions to administer and grade the exam.
• NCBE’s subject matter outlines on the scope of coverage of topics tested within each subject are not detailed enough. Further, the subject matter outlines should indicate the sources of law being tested.
• NCBE should offer more study aids as a low-cost alternative, or supplement, to commercial bar preparation programs.
• NCBE should share more detailed score information with law schools so they can better prepare students.

Next Steps

Results from Phase 1 will be used to inform discussions that are part of Phase 3 on the design of the next generation of the bar examination and the test(s) associated with it. In Phase 2 of the study, a nationwide practice analysis is being conducted to collect information about the importance and frequency of tasks performed by NLLs, as well as the importance of related knowledge, technologies, skills, abilities, and other characteristics necessary for competent entry-level practice. The TTF is committed to continuing to seek input from stakeholders during Phase 3 as it develops recommendations for the next generation of the bar examination and MPRE.