Reliability: arguments for multiple perspectives and potential problems with generalization across studies

Download 197.65 Kb.
View original pdf
Size197.65 Kb.
  1   2   3   4   5   6   7   8   9   ...   23

Kent State University
The present article addresses reliability issues in light of recent studies and debates focused on psychometrics versus datametrics terminology and reliability generalization
(RG) introduced by Vacha-Haase. The purpose here was not to moderate arguments presented in these debates but to discuss multiple perspectives on score reliability and how they may affect research practice, editorial policies, and RG across studies. Issues of classical error variance and reliability are discussed across models of classical test theory, generalizability theory, and item response theory. Potential problems with RG
across studies are discussed in relation to different types of reliability, different test forms, different number of items, misspecifications, and confounding independent variables in a single RG analysis.
The editorial policies of Educational and Psychological Measurement
(EPM) have become a focal point of recent studies and debates related to important issues of reliability, effect size, and confidence intervals in reporting research. The purpose of this article was not to reiterate or moderate datametrics-psychometrics arguments presented in these debates (e.g.,
Thompson, 1994; Sawilowski, 2000; Thompson & Vacha-Haase, 2000) but to discuss issues concerning the assessment of score reliability and how they may affect (current or future) research practice, editorial policies, and reliability generalization (RG) across studies introduced by Vacha-Haase
(1998). The discussion is organized in three parts. The first part presents multiple perspectives on classical standard error of measurement (SEM) and reli-
I would like to thank the two anonymous reviewers and an acting editor for their helpful comments and suggestions. Correspndence regarding this article should be addressed to Dimiter
Dimitrov, PhD, 507 White Hall, College of Education, Kent State University, Kent, OH Educational and Psychological Measurement, Vol. 62, No. 5, October 2002 783-801
DOI: 10.1177/001316402236878
© 2002 Sage Publications 783

ability across traditional and modern measurement models. The second part addresses potential problems with RG across studies in light of issues discussed in the first part. The conclusion summarizes major points of the discussion in the first two parts in an attempt to provide some insight for the practice of substantive research and RG across studies.
In response to EPM editorial policies stating that scores, not tests, are reliable (Thompson, 1994; Thompson & Daniel, 1996), Sawilowski (addressed problems inherent with datametrics from the perspectives of an historical review of reliability terminology and textbook treatments of reliability. He recognized that statements related to reliability of the test" are not sufficiently informative because reliability paradigms and their coefficients are simply not interchangeable ( p. 159). He indicated, however, that reliability estimates should not be tied only to the data at hand:
Indeed, the purpose for using a nationally representative scientifically selected random sample when conducting reliability studies during the test construction process is to obtain the reliability estimate of a test for general purposes. In my view, authors ought always to report these reliability coefficients from manuals and other sources along with the reliability estimate obtained from and a description of, the researcher’s own sample. (p. 170) This point of view, based on numerous treatments of reliability (e.g.,
Crocker & Algina, 1986; Goodenough, 1949; Gronlund, 1988; Nunnally Bernstein, 1994; Payne, 1975; Suen, 1990), is also represented by Traub
(1994) who noted that the usual reliability experiments provide a sample

Share with your friends:
  1   2   3   4   5   6   7   8   9   ...   23

The database is protected by copyright © 2017
send message

    Main page
mental health
health sciences
gandhi university
Rajiv gandhi
Chapter introduction
multiple choice
research methods
south africa
language acquisition
Relationship between
qualitative research
literature review
Curriculum vitae
early childhood
relationship between
Masaryk university
nervous system
Course title
young people
Multiple choice
bangalore karnataka
state university
Original article
academic performance
essay plans
social psychology
psychology chapter
Front matter
United states
Research proposal
sciences bangalore
Mental health
compassion publications
workplace bullying
publications sorted
comparative study
chapter outline
mental illness
Course outline
decision making
sciences karnataka
working memory
Literature review
clinical psychology
college students
systematic review
problem solving
research proposal
human rights
Learning objectives
karnataka proforma