Our April 19 blog summarized work being done to develop the test content specifications (TCS) for the new bar exam, focusing mostly on how that work will get accomplished and who will do it—the stakeholders and subject-matter experts who make up the Content Scope Committee. This blog takes a closer look at the purpose of TCS, what they look like, and why they are important.
TCS describe the key properties of an examination. While they document the content to be covered, they also describe the amount of emphasis allocated to each content area, the assessment format, and other features. TCS go by many names, including test blueprints, content outlines, and test plans. They are typically developed by subject-matter experts who have at their disposal numerous documents and reports (e.g., empirical practice analysis, focus group reports, subject-matter surveys, and existing outlines).
What is the purpose of TCS?
The ultimate goal of TCS is to ensure the content validity and fairness of every test form by providing a blueprint for test development. To accomplish this, TCS have multiple functions that serve different audiences. First, they guide the efforts of item drafters by specifying the practice-relevant content that is eligible for inclusion on an exam. Second, TCS are used by editors and psychometricians to assemble test forms that are comparable in content and difficulty—this helps ensure continuity in test score meaning over time. Third, they inform the users of test scores (e.g., licensing agencies) of exam content so that they can confirm that exam is consistent with their needs. Fourth, students and educators rely on TCS to prepare for the exam. Finally, TCS sometimes provide the basis for performance feedback on different sections of an exam (e.g., you scored at the 85th percentile on section A but at the 45th percentile on section B) in those circumstances for which the sections are sufficiently long to allow for meaningful feedback.
What do TCS typically look like?
The TCS for many high-stakes examinations appear as an outline of topics or skills to be covered. Although it is common for most outlines to run a few or several pages, in recent years they have grown longer and more complex for some testing programs. A format known as the content-by-process matrix is one common framework for organizing test content. A simple version of a content-by-process matrix for a course in US Government might look like this:
Hypothetical Test Content Specification for High-School Course in US Government
Topic
Cognitive Skill: Remember & Understand
Cognitive Skill: Analyze & Apply
Cognitive Skill: Evaluate & Create*
# of Test Items
Origins of US Government
5
4
1
10
The US Constitution
3
9
3
15
The Three Branches of Government
4
11
5
20
State and Local Governments
3
1
1
5
# of Test Items
15
25
10
50 Total
* Note: Test items for Evaluate & Create may include essay questions.
Tables such as this are an informative way to summarize test content. This particular example suggests that of the four topics listed, the Three Branches of Government is the most important, at least in terms of the number of test items allocated to it (20 items). It also implies that the skills of Analyze & Apply will receive more emphasis than the other skills. But, as the footnote to the table indicates, Evaluate & Create will include some essay items, which take more time for students to complete and might receive more weight in grading. So, this example also illustrates that just specifying the number of test items may not be enough; to be fully useful, TCS should also say something about the format of the test items.
TCS for the next generation of the bar exam
NCBE’s Content Scope Committee has just begun work on the TCS for the next generation of the bar exam. Although the final document could be cast as a matrix like that above, it may very well be set out in other formats. Regardless of the layout, we are considering the following elements for possible inclusion:
- An outline of the eight Foundational Concepts & Principles (FC&P) subjects and the topics and subtopics to be covered within each FC&P. The eight FC&P are Civil Procedure, Contracts, Evidence, Torts, Business Associations, Constitutional Law, Criminal Law, and Real Property.
- Sources of law. These will be folded into the document to clarify what is considered to be the controlling authority and, where appropriate, to clarify the breadth and depth of coverage for each FC&P.
- An outline of the seven Foundational Skills (FS) domains that includes skill definitions and examples of practice activities that represent each skill. The seven FS domains are Legal Research, Legal Writing, Issue Spotting and Analysis, Investigation and Evaluation, Client Counseling and Advising, Negotiation and Dispute Resolution, and Client Relationship and Management.
- The degree of emphasis allocated to each FC&P category and FS domain. Degree of emphasis could be expressed as the number of test items, the amount of testing time, or the allocated scoring weight.
- The mix of test-item formats to be included. The next generation of the bar exam will include the formats required to provide all examinees with sufficient opportunity to demonstrate what they know and can do. The formats will include, but will not be limited to, multiple-choice and essay-type questions. Both item sets and stand-alone items will be used.
From TCS to test design
A related tool used by testing organizations is a test design. While the test design includes the TCS, it also documents all other operational features of a testing program including procedures such as the frequency and duration of test administration, mode of delivery, and the allocation of pretest items. We’ll take up the topic of test design in a separate blog.
