KUKU

KU ScholarWorks

  • myKU
  • Email
  • Enroll & Pay
  • KU Directory
    • Login
    View Item 
    •   KU ScholarWorks
    • Dissertations and Theses
    • Dissertations
    • View Item
    •   KU ScholarWorks
    • Dissertations and Theses
    • Dissertations
    • View Item
    JavaScript is disabled for your browser. Some features of this site may not work without it.

    Technology Enabled Assessments: An Investigation of Scoring Models for Scaffolded Tasks

    Thumbnail
    View/Open
    Nash_ku_0099D_12028_DATA_1.pdf (2.462Mb)
    Issue Date
    2012-05-31
    Author
    Nash, Brooke L.
    Publisher
    University of Kansas
    Format
    192 pages
    Type
    Dissertation
    Degree Level
    Ph.D.
    Discipline
    Psychology & Research in Education
    Rights
    This item is protected by copyright and unless otherwise specified the copyright of this thesis/dissertation is held by the author.
    Metadata
    Show full item record
    Abstract
    While significant progress has been made in recent years on technology enabled assessments (TEAs), including assessment systems that incorporate scaffolding into the assessment process, there is a dearth of research regarding psychometric scoring models that can be used to fully capture students' knowledge, skills and abilities as measured by TEAs. This investigation provides a comparison of seven scoring models applied to an operational assessment system that incorporates scaffolding into the assessment process and evaluates student ability estimates derived from those models from a validity perspective. A sequential procedure for fitting and evaluating increasingly complex models was conducted. Specifically, a baseline model that did not account for any scaffolding features in the assessment system was established and compared to three additional models that each accounted for scaffolding features using a dichotomous, a polytomous and a testlet model approach. Models were compared and evaluated against several criteria including model convergence, the amount of information each model provided and the statistical relationships between scaled scores and a criterion measure of student ability. Based on these criteria, the dichotomous model that accounted for all of the scaffold items but ignored local dependence was determined to be the optimal scoring model for the assessment system used in this study. However, if the violation against the local independence assumption is deemed unacceptable, it was also concluded that the polytomous model for scoring these assessments is a worthwhile and viable alternative. In any case, the scoring models that accounted for the scaffolding features in the assessment system were determined to be better overall models than the baseline model that did not account for these features. It was also determined that the testlet model approach was not a practical or useful scoring option for this assessment system. Given the purpose of the assessment system used in this study, which is a formative tool that also provides instructional opportunities to students during the assessment process, the advantages of applying any of these scoring models from a measurement perspective may not justify the practical disadvantages. For instance, a basic percent correct score may be completely dependent on the specific items that a student took but it is relatively simple to understand and compute. On the other hand, scaled scores from these scoring models are independent of the items from which they were calibrated from, but ability estimates are more complex to understand and derive. As the assessment system used in this study is a low stakes environment that is mostly geared towards learning, the benefits of the scoring models presented in this study need to be weighed against the practical constraints within an operational context with respect to time, cost and resources.
    URI
    http://hdl.handle.net/1808/10013
    Collections
    • Dissertations [4475]
    • Education Dissertations and Theses [1065]

    Items in KU ScholarWorks are protected by copyright, with all rights reserved, unless otherwise indicated.


    We want to hear from you! Please share your stories about how Open Access to this item benefits YOU.


    Contact KU ScholarWorks
    785-864-8983
    KU Libraries
    1425 Jayhawk Blvd
    Lawrence, KS 66045
    785-864-8983

    KU Libraries
    1425 Jayhawk Blvd
    Lawrence, KS 66045
    Image Credits
     

     

    Browse

    All of KU ScholarWorksCommunities & CollectionsThis Collection

    My Account

    LoginRegister

    Statistics

    View Usage Statistics

    Contact KU ScholarWorks
    785-864-8983
    KU Libraries
    1425 Jayhawk Blvd
    Lawrence, KS 66045
    785-864-8983

    KU Libraries
    1425 Jayhawk Blvd
    Lawrence, KS 66045
    Image Credits
     

     

    The University of Kansas
      Contact KU ScholarWorks
    Lawrence, KS | Maps
     
    • Academics
    • Admission
    • Alumni
    • Athletics
    • Campuses
    • Giving
    • Jobs

    The University of Kansas prohibits discrimination on the basis of race, color, ethnicity, religion, sex, national origin, age, ancestry, disability, status as a veteran, sexual orientation, marital status, parental status, gender identity, gender expression and genetic information in the University’s programs and activities. The following person has been designated to handle inquiries regarding the non-discrimination policies: Director of the Office of Institutional Opportunity and Access, IOA@ku.edu, 1246 W. Campus Road, Room 153A, Lawrence, KS, 66045, (785)864-6414, 711 TTY.

     Contact KU
    Lawrence, KS | Maps