An Investigation of the Quality of Student-Developed Surveys and Ratings Scales and Psychometric Reporting Practices in Doctoral Dissertations

Abstract

Doctoral dissertation research has been criticized for its quality and contribution to scholarly research. Commonalities between doctoral research and reviews of published articles indicate a lack of psychometric reporting practices of those utilizing instrumentation. Surveys, specifically those that are self-developed, have not been examined in doctoral research. Multiple sources advise researchers who create their own surveys for data collection to follow specific item writing and rating scale development guidelines. This previous research led to the investigation of student-developed surveys and rating scales in doctoral dissertations, and the psychometric reporting practices of the students, to identify trends in graduate research. Two-hundred forty-six doctoral dissertations were examined, which included the use of 280 self-developed surveys. Specific guidelines were created for assessing the survey characteristics and students’ reporting of psychometric properties. The survey items and rating scale characteristics were considered favorable; the authors mostly adhered to survey development guidelines. The frequency of students who validated their surveys was superb. However, the lack of reliability reporting by students calls into question not only the knowledge and experience with reliability methods of the students and their committee members, but also the psychometric training students experience in graduate school.