| The Role of Probability-Based Inference in an Intelligent Tutoring System |
OCT 95 |
50 pages |
| Authors:
Robert J. Mislevy; Drew H. Gitomer; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
 | Probability-based inference in complex networks of interdependent variables is an active topic in statistical research, spurred by such diverse applications as forecasting, pedigree analysis, troubleshooting, and medical diagnosis. This paper concerns the role of Bayesian inference networks for updating student models in intelligent tutoring systems (ITSs). Basic concepts of the approach are briefly reviewed, but the emphasis is on the considerations that arise when one attempts to operationalize the abstract ... |
|
| The Role of Probability-Based Inference in an Intelligent Tutoring System |
OCT 95 |
49 pages |
| Authors:
Robert J. Mislevy; Drew H. Gitomer; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| Probability-based inference in complex networks of interdependent variables is an active topic in statistical research, spurred by such diverse applications as forecasting, pedigree analysis, troubleshooting, and medical diagnosis. This paper concerns the role of Bayesian inference networks for updating student models in intelligent tutoring systems (ITSs). Basic concepts of the approach are briefly reviewed, but the emphasis is on the considerations that arise when one attempts to operationalize the abstract ... |
|
| The Role of Probability-Based Inference in an Intelligent Tutoring System |
OCT 95 |
50 pages |
| Authors:
Robert J. Mislevy; Drew H. Gitomer; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| Probability-based inference in complex networks of interdependent variables is an active topic in statistical research, spurred by such diverse applications as forecasting, pedigree analysis, troubleshooting, and.medical diagnosis. This paper concerns the role of Bayesian inference networks for updating student models in intelligent tutoring systems (ITSs). Basic concepts of the approach are briefly reviewed, but the emphasis is on the considerations that arise when,one attempts to operationalize the abstract framework of ... |
|
| Evidence and Inference in Educational Assessment |
FEB 95 |
80 pages |
| Authors:
Robert J. Mislevy; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| Educational assessment concerns inference about students' knowledge, skills, and accomplishments. Because data are never so comprehensive and unequivocal as to ensure certitude, test theory evolved in part to address questions of weight, coverage, and import of data. The resulting concepts and techniques can be viewed as applications of more general principles for inference in the presence of uncertainty. Issues of evidence and inference in educational assessment are discussed from this ... |
|
| Test Theory Reconceived |
JAN 95 |
59 pages |
| Authors:
Robert J. Mislevy; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| Educational test theory consists of statistical and methodological tools to support inference about examinees' knowledge, skills, and accomplishments. The evolution of test theory has been shaped by the nature of users' inferences, which until recently, have been framed almost exclusively in terms of trait and behavioral psychology. Progress in the methodology of test theory enabled users to extend the range of inference, sharpen the logic, and ground their interpretations more ... |
|
| Can There Be Reliability Without "Reliability?" |
OCT 94 |
24 pages |
| Authors:
Robert J. Mislevy; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| A recent article by Pamela Moss asks the title question, 'Can there be validity without reliability?' If by reliability we mean only KR-2O coefficients or inter-rater correlations, the answer is yes. Sometimes these particular indices for evaluating evidence suit the problem we encounter; sometimes they don't. If by reliability we mean credibility of evidence, wehre credibility is defined as 'appropriate to the intended inference, the answer is no, we cannot ... |
|
| Virtual Representation of IID Observations in Bayesian Belief Networks |
APR 94 |
20 pages |
| Authors:
Robert J. Mislevy; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| Local computation for updating Bayesian belief networks proceeds in the context of a join tree, consisting of subsets of interrelated variables (cliques) joined by their intersection sets in a singly-connected graphical structure. When multiple independent and identically-distributed (IID) observations of a variable can be made, identically structured cliques corresponding to each potential observation appear as terminal nodes in the join tree. This note shows how it is possible to absorb ... |
|
| Information-Decay Pursuit of Dynamic Parameters in Student Models |
APR 94 |
34 pages |
| Authors:
Robert J. Mislevy; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| Learning processes are often analyzed using a model with a structure that is assumed to apply to all points in time, but with one or more parameters that may change over time. In some applications, such as learning theory, the goal is to model the character of change itself; observations at all time points are used to estimate, for example, learning curves. In other applications, such as tutoring systems, the ... |
|
| Dealing with Uncertainty about Item Parameters: Expected Response Functions |
APR 94 |
49 pages |
| Authors:
Robert J. Mislevy; Marilyn S. Wingersky; Kathleen M. Sheehan; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| It is a common practice in item response theory (IRT) to treat estimates of item parameters, say B, as if they were the known, true quantities, B. However, ignoring the uncertainty associated with item parameters can lead to biases and over-confidence in subsequent inferences such as ability estimation, especially when item-calibration samples are small. This paper demonstrates how to incorporate uncertainty about B with Lewis's expected response functions (ERFs) , ... |
|
| Diagnostic Assessment of Troubleshooting Skill in an Intelligent Tutoring System |
MAR 94 |
56 pages |
| Authors:
Drew H. Gitomer; Linda S. Steinberg; Robert J. Mislevy; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| This paper lays out the rationale and implementation of student modeling and updating in the HYDRIVE intelligent tutoring system (ITS) for aircraft hydraulic systems. An epistemic level of modeling concerns the plans and goals students are using to guide their problem-solving, as inferred from specific actions in specific contexts. These results update a student model constructed around more broadly defined aspects of system understanding, strategic knowledge, and procedural skills. Meant ... |
|
| Probability-Based Inference in Cognitive Diagnosis |
FEB 94 |
58 pages |
| Authors:
Robert J. Mislevy; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| Recent developments in cognitive psychology suggest models for knowledge and learning that often fall outside the realm of standard test theory. This paper concerns probability-based inference in terms of such models. An approach utilizing Bayesian inference networks is outlined. Basic ideas of structure and computation in inference networks are discussed, and illustrated with an example from the domain of mixed-number subtraction. Bayesian inference, Belief nets, Cognitive diagnosis, Psychology, Educational measurement, ... |
|
| How to Equate Tests with Little or No Data |
FEB 92 |
51 pages |
| Authors:
Robert J. Mislevy; Kathleen M. Sheehan; Marilyn Wingersky; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| Standard procedures for equating tests, including those based on item response theory (IRT), require item responses from large numbers of examinees. Such data may not be forthcoming for reasons theoretical, political, or practical. Information about items' operating characteristics may be available from other sources, however such as content and format skills and strategies required to solve them. This paper shows how, in the IRT framework, collateral information about items can ... |
|
| The Variance of Rasch Ability Estimates from Partially-Known Item Parameters |
JAN 92 |
20 pages |
| Authors:
Robert J. Mislevy; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| A closed-form approximation is given for the variance of examinee proficiency estimates in the Rasch model for dichotomous items, under the condition that only estimates, rather than true values, of item difficulty parameters are available. The term that must be added to the usual response- sampling variance is inversely proportional to both the number of examinees in the item calibration sample and the length of the test. Illustrative numerical values ... |
|
| Probability-Based Inference in a Domain of Proportional Reasoning Tasks |
JAN 92 |
62 pages |
| Authors:
Anne Beland; Robert J. Mislevy; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| Educators and psychologists are increasingly interested in modelling the processes and knowledge structures by which people learn and solve problems. Progress has been made in developing cognitive models in several domains, and in devising observational settings that provided clues about subject's cognition from this perspective. Less attention has been paid to procedures for inference or decision making with such information, given that it provides only imperfect information about cognition in ... |
|
| Toward a Test Theory for assessing Student Understanding |
APR 91 |
44 pages |
| Authors:
Robert J. Mislevy; Kentaro Yamamoto; Steven Anacker; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| The view of learning that underlies standard test theory is inconsistent with the view rapidly emerging from cognitive and educational psychology. Learners become more competent not simply by learning more facts and skills, but by reconfiguring their knowledge; by 'chunking' information to reduce memory loads; and by developing strategies and models that help them discern when and how facts and skills are important. Neither classical test theory nor item response ... |
|
| New Views of Student Learning: Implications for Educational Measurement |
MAR 91 |
48 pages |
| Authors:
Geofferey N. Masters; Robert J. Mislevy; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| Recent research in cognitive psychology has drawn attention to the important role that students' personal understandings and representations of subject matter play in the learning process. This paper briefly reviews some of this research, and contrasts the kind of learning that results in an individual's changed conception or view of a phenomenon with the more passive, additive kind of learning assessed by most traditional achievement tests. To be consistent with ... |
|
| Resolving Mixtures of Strategies in Spatial Visualization Tasks |
JUL 90 |
63 pages |
| Authors:
Robert J. Mislevy; Marilyn S. Wingersky; Sidney H. Irvine; Peter L. Dann; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| The models of standard test theory, having evolved under a trait- oriented psychology, do not reflect the knowledge structures and the problem- solving strategies now seen as central to understanding performance and learning. In some applications, however, key qualitative distinctions among persons as to structures and strategies can be expressed through mixtures of test theory models, drawing upon substantive theory to delineate the components of the mixture. This approach is ... |
|
| Foundations of a New Test Theory |
OCT 89 |
44 pages |
| Authors:
Robert J. Mislevy; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| It is only a slight exaggeration to describe the test theory that dominates educational measurement today as the application of twentieth century statistics to nineteenth century psychology. Sophisticated estimation procedures, new techniques for missing-data problems, and theoretical advances into latent-variable modeling have appeared--all applied with psychological models that explain problem-solving ability in terms of a single, continuous variable. This caricature suffices for many practical prediction and selection problems because it ... |
|
| The Role of Collateral Information about Examinees in Item Parameter Estimation |
SEP 88 |
57 pages |
| Authors:
Robert J. Mislevy; Kathleen M. Sheehan; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| Standard procedures for estimating item parameters in item response theory (IRT) ignore collateral information that may be available about examinees, such as their standing on demographic and educational variables. This paper describes circumstances under which collateral information about examinees may be used to make inferences about item parameters more precise, and circumstances under which it must be used to obtain correct inferences. |
|
| Randomization-Based Inferences about Latent Variables from Complex Samples |
SEP 88 |
76 pages |
| Authors:
Robert J. Mislevy; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| Standard procedures for drawing inferences from complex samples do not apply when the variable of interest theta cannot be observed directly, but must be inferred from the values of secondary random variables that depend on theta stochastically. Examples are examinee proficiency variables in item response theory models and class memberships in latent class models. This paper uses Rubin's multiple imputation approach to approximate sample statistics that would have been ... |
|
| Exploiting Collateral Information in the Estimation of Item Parameters |
SEP 88 |
36 pages |
| Authors:
Robert J. Mislevy; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| When using item response theory (IRT) models in educational and psychological measurement, it is standard practice to estimate the operating characteristics of test items from examinees' item responses alone. This is the final report of a project that employed Bayesian and empirical Bayesian methods to exploit additional information that is often available about test items (e.g. , format, content, or cognitive processing requirements) or about examinees (e. g., educational background ... |
|
| Inferring Examinee Ability When Some Item Responses Are Missing |
SEP 88 |
80 pages |
| Authors:
Robert J. Mislevy; Pao-Kuei Wu; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| The basic equations of item response theory (IRT) provide a foundation for inferring examinees' abilities and items' operating characteristics from observed responses. In practice, though, examinees will usually not have provided a response to every available item--for reasons that may or may not have been intended by the test administrator, and that may or may not be related to examinee ability. The mechanisms that produce missing data must be taken ... |
|
| Some Consequences of the Uncertainty in IRT (Item Response Theory) Linking Procedures |
JUL 88 |
41 pages |
| Authors:
Kathleen M. Sheehan; Robert J. Mislevy; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| In many practical applications of item response theory, the parameters of overlapping subsets of test items are estimated from different samples of examinees. A linking procedure is then employed to place the resulting item parameter estimates onto a common scale. It is standard practice to ignore the uncertainty associated with the linking step when drawing inferences that involve items from different subsets, a situation that arises, for example, in the ... |
|
| The Information Matrix in Latent-Variable Models |
APR 88 |
38 pages |
| Authors:
Robert J. Mislevy; Kathleen M. Sheehan; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| The information matrix for the parameters in a latent-variable model is bounded from above by the information that would obtain if the values of the latent variables could also be observed. The difference is the missing information. This paper discusses the structure of the information matrix, and characterizes the degree to which missing information can be recovered by exploiting collateral variables for respondents. The results are illustrated with data from ... |
|
| Modeling Item Responses When Different Subjects Employ Different Solution Strategies |
OCT 87 |
|
| Authors:
Robert J. Mislevy; Norman Verhelst; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
 | A model is presented for item responses when different examinees employ different strategies to arrive at their answers, and when only those answers, not choice of strategy or subtask results, can be observed. Using substantive theory to differentiate the likelihoods of response vectors under a fixed set of solution strategies, the authors model responses in terms of item parameters associated with each strategy, proportions of the population employing each, and ... |
|
| Exploiting Auxiliary Information about Items in the Estimation of Rasch Item Difficulty Parameters |
JUL 87 |
|
| Authors:
Robert J. Mislevy; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| Standard procedures for estimating th item parameters in IRT models make no use of auxiliary information about test items, such as their format or content, or the skills they require for solution. This paper describes a framework for exploiting this information, thereby enhancing the precision and stability of item parameter estimates and providing diagnostic information about items' operating characteristics. The principles are illustrated in a context for which a relatively ... |
|
| Exploiting Auxiliary Information about Examinees in the Estimation of Item Parameters |
MAY 86 |
|
| Authors:
Robert J. Mislevy; EDUCATIONAL TESTING SERVICE PRINCETON NJ
|
| A pervasive problem in item response theory (IRT) is the difficulty of simultaneously estimating large numbers of parameters from limited data. Even large samples of examinees may not eliminate the problem when each examinee responds to only a few items, as in educational assessment and adaptive testing. The precision of item parameter estimates can be increased by taking advantage of dependencies between the latent proficiency variable and auxiliary examinee variables ... |
|
Reports by Author
