TDC Meeting Evaluation Responses

Evaluation Question	Responses
Meeting #1
How prepared did you feel for the discussions during the meeting based on your experience with the bar exam and the advance materials?
Very prepared	12
Somewhat prepared	4
Not prepared	0
How would you rate the time allocated to discussing each test design topic?
More than enough time	1
Right amount of time	13
Not enough time	1
Meeting #2
How would you rate the time allocated to discussing the feedback on each model and delineating the final expectations?
More than enough time	0
Right amount of time	9
Not enough time	3
Did not attend the August 4th Meeting	3
Overall Process
Please rate the success of the process overall (Meeting #1, Review of Models, Meeting #2) in soliciting and distilling the input from TDC members into recommendations for the design of the exam
Very successful	1
Successful	9
Not successful	1
Please rate your opportunity to contribute to this process
Plenty of opportunity	8
Sufficient opportunity	5
Not enough opportunity	2

 

TDC Meeting Evaluation Comments

• I had a difficult time answering the “amount of time” questions. I felt there were some topics we spent too much time on (for example, revisiting the work done by committees that came before us) and others that did not receive enough discussion. In the end, I trusted the process and the leaders to make sure we spent the time where it was necessary.
• I valued the input of so many different points of view. No matter what comes of this process, I cannot honestly say that I did not have the opportunity to share my perspective. Having said that, it felt at times like there were some voices that dominated the conversation and perhaps the frequency and length of those comments seemed to tilt the discussion that direction. Those strong voices served as advocates for their positions, but length and volume should not take supplant those who spoke quietly or concisely.
• Finally, thank you for the opportunity to be part of something that is so meaningful. I really appreciated it!
• I think the process was generally good. At some points, it did feel like the topics were pre-determined to the extent where some additional ideas may not have been given enough time. For example, I think there should have been a decision made about whether to recommend a closed- or open-book bar exam. I think one of the better models that I have seen in other places is the open-book concept, where a base knowledge of the core doctrinal principles would be necessary in order to successfully navigate the exam, even while using study materials. We brushed up against that idea at times, but that is the big change that I would have advocated to address what I perceive are some of the biggest weaknesses/criticisms of the exam. Those weaknesses/criticisms would be 1) that the exam is more about memorization at this point than the actual practice of law consists of, 2) it takes too long to study for the exam by having to memorize too many legal nuances, which means we cannot offer the exam any earlier than we currently do based upon typical graduation dates, and 3) there is a real disadvantage to people who cannot afford expensive bar prep courses and to those who cannot afford to not work at least part time. The only other criticism that I have is that we too often got sidetracked talking about the bar exam’s effect on the legal education world as opposed to what a test that measures minimum competency to practice law should consist of and look like. While I appreciate the concerns of the legal educators, and it is certainly something to consider when ultimately crafting the exam of the future, it felt like that was the issue that we discussed the most.
• I thought that the overall process was great, but that we were rushed at the end of the August 4 meeting.
• Individuals from the law schools dominated the conversation and often got way off topic. They were more concerned about how any changes would impact THEM, as opposed to deciding what changes, if any, to the bar exam would further the goal of ensuring that newly licensed lawyers had minimum competence in legal knowledge and skills to enter the practice of law.
• This was a very educational and worthwhile experience. The opportunity to discuss these issues with people of varied roles in the process was enlightening. I hope to have the opportunity to be involved in something of this nature again.
• I appreciated this opportunity. I learned so much from listening to the other participants. And I respect the time and resources that are being applied to this project.
• I would have liked to see examples of what was meant by the new style of questions such as case study with selected response, what the short answer would look like in order to be computer-scored, etc. Some educators think short answer is a fill-in-the-blank style question, others a sentence, and others a paragraph. I did have a suggestion for finding out when your questions go to far. I never feel the person creating the question can truly tell. You need to ask the people grading them. So, do a survey of actual graders over this July, September, and October’s essays and MPTs asking about each essay and each subquestion on the essay—inquiring how students did, how grader felt if it was in the wheelhouse of the basics or beyond, etc. As one of the graders noted, the first part of the Trusts essay was beyond even him, but then it was fine. That’s the information you are going to need. Property professors should have known that the Fair Housing Act is not taught in every law school’s first year property classes. Far from being basic. Then, that was the whole question—average student totally screwed. As least they all were.
• These calls happened during very busy times in our office. I feel bad that I was not able to verbally participate to the level I wished to contribute.
• My only complaint was that the group was too big.
• I liked how the differing positions were coalesced into a new proposal by finding the similarities in each. I thought the testing professionals were really good.
• The large group discussion at the 8/4 meeting was more challenging and less productive than the earlier smaller group discussions. Participants spent too much time on 8/4 responding to each other’s assumptions and beliefs rather than discussing the options and decisions before the committee.
• This is a challenging survey to fill out. I appreciate the hard work to include a variety of voices and perspectives. And I think the efforts at facilitating and including those perspectives were serious and sincere, and I learned a good deal over the course of the several days and appreciated the opportunity to be included. But at an important structural level, the purposes of the bar exam and its potential effects on legal education, and, frankly, the purposes of law school, did not, in my opinion, get a deep enough airing, and yet they must significantly inform test design. I suspect there would be a significant difference in perspective about the appropriate design (and timing) of the bar exam among deans from different ‘tiers’ of schools—indeed we saw that play out in the meeting, and we would likely see something similar with broader engagement. Several of us—but certainly a minority—were very concerned about the effects that certain changes would have on legal education. Others—not deans of law schools!—didn’t appear to care one whit for that concern. Many participants thought that everything important ought to be tested, somehow, on the exam—but there was little engagement (nor do most of us have the expertise) to understand what can be tested most effectively and reliably. How one thinks about these questions also depends greatly on priors: is the bar exam a critical tool for weeding out hordes of unqualified law graduates that would otherwise prey on the public?; is it an important moment forcing qualified attorneys to consolidate and synthesize knowledge in a high-stress situation?; is it a check to ensure that a very small number of unqualified attorneys are located before they begin to practice? Should we be more concerned with type one or type two errors? Should we be more concerned for verisimilitude on the exam with the tasks newly licensed attorneys engage with in practice, or ought we to care more about test reliability across sittings/graders even if that pushes towards multiple choice exams? In any event, I hope that the materials about what emerged from this set of meetings will make clear and acknowledge the lack of consensus on a number of these important questions, as well as with regard to several of the recommendations.