PubMed · 40168289
Assessing construct reliability through open-ended survey response analysis.
Abstract
Online surveys often include quantitative attention checks, but inattentive participants might also be identified using their qualitative responses. We used the software Turnitin™ to assess the originality of open-ended responses in four mixed-method surveys that included validated multi-item rating scales (i.e., constructs). Across surveys, 18-35% of participants (n =  3,771) were identified as having copied responses from online sources. We assessed indicator reliability and internal consistency reliability and found that both were lower for participants identified as using copied text versus those who wrote more original responses. Those who provided more original responses also provided more consistent responses to the validated scales, suggesting that these participants were more attentive. We conclude that this process can be used to screen open-ended responses from online surveys. We encourage future research to replicate this screening process using similar tools, investigate strategies to reduce copying behaviour, and explore the motivation of participants to search for information online, including what sources they find compelling.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
Katherine E Koralesky, Marina A G von Keyserlingk, Daniel M Weary. 2025-04-01. Assessing construct reliability through open-ended survey response analysis.. https://doi.org/10.1371/journal.pone.0320570. https://searchpubmed.org/record/pubmed%3A40168289/assessing-construct-reliability-through-open-ended-survey-response-analysis
Citations retain the original authors and DOI and include this site’s record URL. If this discovery service contributed to your research, you can also cite Search PubMed by Indiciae: BibTeX · RIS.