Overview and Next Steps
The purpose of the Testing Task Force’s study over the past three years has been to gather the necessary research to inform how the bar exam should be changed to adapt to a changing profession. The study has been conducted with transparency and has included input from stakeholders each step of the way.
For Phase 3 of the Task Force’s work, two committees were convened to discuss test blueprint and design issues by working from the qualitative and quantitative data that were compiled in Phase 1 (stakeholder listening sessions) and Phase 2 (nationwide practice analysis). The charge of the Blueprint Development Committee (BDC) was to help determine what content should be tested on the bar exam, while the role of the Test Design Committee (TDC) was to recommend how that content should be assessed. The BDC consisted of newly licensed and experienced practitioners who provided input on exam content using the Phase 2 study results and applying their professional judgment and experience. The TDC was composed of legal educators and bar admission representatives who provided input on an effective structure and design for the exam. The TDC’s work was guided by the Phase 1 study results and by the professional judgment and experience of committee members in educating law school students and admitting newly licensed lawyers (NLLs) to the bar. This Phase 3 report summarizes the work completed by the BDC and the TDC.
Using the results from the BDC and TDC meetings, the Task Force is formulating a set of preliminary recommendations for the content and design of the next generation of the bar exam. The starting point for developing any exam is to consider the purpose of the test and the claim(s) to be made about examinee performance. Early in the study, the Task Force adopted the following claim for the intended use of scores from the bar exam: To protect the public by helping to ensure that those who are newly licensed possess the minimum knowledge and skills to perform activities typically required of an entry-level lawyer. This claim guided the work of the BDC and TDC and will serve as foundation for the Task Force’s recommendations for the next generation of the bar exam.
Beyond fidelity to the claim, the content and design of the exam must be constrained by psychometric requirements relating to reliability, validity, and fairness. Therefore, the Task Force has consulted, and will continue to consult, with its internal and external assessment experts in formulating recommendations that satisfy the Standards for Educational and Psychological Testing (AERA, APA, NCME, 2014). The other key objectives guiding the Task Force as it develops recommendations include
- ensuring that the depth and breadth of the content tested is carefully aligned with minimum competence for entry-level practice;
- increasing emphasis on assessment of lawyering skills;
- adopting an integrated approach to assessing knowledge and skills;
- continuing to exercise vigilance to prevent bias and ensure fairness and accessibility for all examinees;
- reducing the length of the exam to the extent possible without sacrificing reliability and validity of score interpretations;
- administering the exam by computer either at test centers managed by vendors or on examinees’ laptops in jurisdiction-managed testing facilities;
- keeping the exam affordable so that cost does not pose a barrier to entering the profession;
- reducing the time between exam administration and announcement of results through greater efficiencies in scoring and grading while maintaining the high quality of processes used; and
- maintaining the benefits of score portability realized through use of the Uniform Bar Exam.
The Task Force’s preliminary recommendations will be shared with bar admission authorities and the legal academy for comment before recommendations are finalized by the Task Force for submission to NCBE’s Board of Trustees in January 2021.
Blueprint Development Committee Meeting
The TTF recruited 17 practicing lawyers to participate as panelists on the BDC; 14 of the panelists were female and 10 were people of color. In total, the panelists practiced in 13 jurisdictions and across a range of practice settings (private law firm, government, nonprofit organization, legal services/public interest, judicial law clerk, and in-house counsel). Each panelist indicated his or her area(s) of practice, which collectively included:
- Administrative Law
- Business Law
- Civil Rights
- Commercial Law
- Criminal Law
- Disability Rights
- Education Law
- Employee Benefits
- Employment Law and Labor Relations
- Estate Planning
- Fair Housing
- Family Law
- Health Care Law
- Insurance Coverage
- Personal Injury
- Professional Regulation
The BDC met from June 29 to July 1, 2020. The meeting was held virtually for five hours each day. Prior to the meeting, each panelist was provided a binder of materials that served as advance readings for the meeting:
- Meeting Agenda
- NCBE Testing Task Force Study Overview
- Testing Task Force Phase 1 Executive Summary
- Testing Task Force Phase 2 Report
Additional materials were provided closer to the meeting date for reference during the meeting:
- Detailed results of Phase 2 Analysis
- Tasks organized by overall frequency of performing with results provided by Practice Cluster
- Knowledge Areas organized by overall importance with results provided by Practice Cluster
- Proposed Blueprint Structure – tasks organized under seven skill domains and knowledge areas
- Information about the structure of the current Uniform Bar Examination (UBE) and Multistate Professional Responsibility Examination (MPRE)
- Current MPRE Subject Matter Outline
Dr. Chad Buckendahl and Dr. Susan Davis-Becker from ACS Ventures LLC (ACS) facilitated the meeting. Staff from NCBE (Kellie Early and Dr. Mark Raymond) and the Chair of the TTF (Hon. Cindy Martin) attended the meeting to observe. Following introductory remarks by Judge Martin, ACS led an orientation that began with a review of the primary claim for the intended use of scores from the bar exam:
To protect the public by helping to ensure that those who are newly licensed possess the minimum knowledge and skills to perform activities typically required of an entry-level lawyer.
ACS then provided an overview of the TTF study, the purpose and function of a test blueprint, and the plan for the three-day meeting. The facilitators walked the panelists through the meeting materials and explained how to interpret the results of the Phase 2 practice analysis. This orientation took the majority of the first day of the meeting.
The general discussion began after the orientation with a review of the job tasks from the practice analysis survey. Specifically, the full list of 179 tasks was reduced to those 136 tasks that were rated as being performed Frequently or Moderately by 50%1 or more of the survey respondents. Further, the tasks identified for review were organized by the TTF and ACS under these seven skill domains:
- Legal Research
- Legal Writing and Drafting
- Client Counseling and Advising
- Issue Spotting and Evaluation
- Investigation and Analysis
- Negotiation and Dispute Resolution
- Client Relationship and Management
The BDC reviewed each task and discussed its relevance to practice by NLLs based on the ratings collected during the practice analysis, including (1) the overall frequency ratings (how frequently the tasks were performed by NLLs), (2) the frequency ratings by Practice Cluster (survey respondents grouped by practice areas), and (3) the frequency ratings by those survey respondents identified as NLLs versus those who were not NLLs. The result of each task-level discussion was a recommendation as to whether the task should be included within that skill domain as being representative of the activities required of NLLs. The BDC also recommended consolidation of some tasks to eliminate overlap or redundancy.
After reviewing all 136 tasks in this manner, the BDC was asked to consider how much emphasis or weight should be given to the seven skill domains on the bar exam, including models of (1) equal weighting for each skill domain, (2) natural weighting, meaning the weight is determined by the number of tasks under each skill domain, or (3) weighting based on the judgments of the BDC. The BDC panelists opted for the third model and applied their judgment to reach consensus on recommended weights for each skill domain. This activity was concluded at the end of the second day.
The third day of the meeting was focused on reviewing the knowledge areas from the practice analysis. The full list of 77 knowledge areas from the practice analysis survey was reduced to 25 by prioritizing those areas that were rated as Important by 50%2 or more of the survey respondents. The BDC reviewed each knowledge area and discussed the relevance to practice by NLLs based on the overall importance ratings, the importance ratings by Practice Cluster, and the importance ratings by those respondents identified as NLLs versus those who were not NLLs. The result of each knowledge area discussion was a recommendation as to whether the area should be included on the bar exam.
After making decisions about what knowledge areas to recommend for inclusion, the BDC considered how much emphasis or weight should be given to each knowledge area on the bar exam. The BDC also considered generally whether knowledge areas should be measured in a content-dependent context (necessary legal resources are not provided) or in a content-independent context (necessary legal resources are provided). There was not enough time at the end of the third day to allow for coming to consensus on weighting or how the knowledge areas should be measured. Therefore, the facilitators introduced the topic and the panelists were sent a post-meeting assignment to provide their recommendations.
Each panelist was also asked to complete an online evaluation of the blueprint development meeting.
In total, the BDC identified 103 tasks from the original list of 136 tasks as representative of the seven skill domains identified for assessment on the bar exam: 9 of the original 136 tasks were consolidated to eliminate redundancy, and 24 tasks were recommended for exclusion, with most of those excluded because the BDC concluded that the tasks exceed the scope of an NLL’s practice. As an example, “Prepare or designate record for appellate or other post-judgmental review” is a task that the BDC recommended for exclusion. The list of tasks that were considered and the results can be found in Appendix A.
Table 1 shows for each skill domain the number of tasks, a general description of the domain, and the recommended weighting. The weighting is shown as the average of the weights recommended by the BDC panelists; a range of roughly 3% around that average is shown in parentheses.
|Description of Domain
|Researching the Law, Written/Reading Comprehension, Critical/Analytical Thinking
|Legal Writing and Drafting
|Written Expression, Critical/Analytical Thinking
|Client Counseling and Advising
|Oral Expression, Oral Comprehension, Cultural Competence, Advocacy, Critical/Analytical Thinking, Problem Solving, Practical Judgment
|Issue Spotting and Evaluation
|Identifying Issues, Observant, Critical/Analytical Thinking
|Investigation and Analysis
|Interviewing/Questioning, Fact Gathering, Cultural Competence, Problem Solving
|Negotiation and Dispute Resolution
|Negotiation Skills/Conflict Resolution, Creativity/Innovation, Expressing Disagreement, Written Expression, Oral Expression, Oral Comprehension, Advocacy, Practical Judgment
|Client Relationship and Management
|Networking and Business Development, Resource Management/ Prioritization, Organization, Strategic Planning, Managing Projects, Achievement/Goal Orientation, Practical Judgment, Decisiveness, Cultural Competence