To be Direct or notReversing Likert Response Format Items

  1. Jaime García-Fernández 1
  2. Álvaro Postigo 1
  3. Marcelino Cuesta 1
  4. Covadonga González-Nuevo 1
  5. Álvaro Menéndez-Aller 1
  6. Eduardo García-Cueto 1
  1. 1 Universidad de Oviedo
    info

    Universidad de Oviedo

    Oviedo, España

    ROR https://ror.org/006gksa02

Revista:
The Spanish Journal of Psychology

ISSN: 1138-7416

Año de publicación: 2022

Número: 25

Páginas: 1-9

Tipo: Artículo

DOI: 10.1017/SJP.2022.20 DIALNET GOOGLE SCHOLAR lock_openAcceso abierto editor

Otras publicaciones en: The Spanish Journal of Psychology

Resumen

Likert items are often used in social and health sciences. However, the format is strongly affected by acquiescence and reversed items have traditionally been used to control this response bias, a controversial practice. This paper aims to examine how reversed items affect the psychometric properties of a scale. Different versions of the Grit-s scale were applied to an adult sample (N = 1,419). The versions of the scale had either all items in positive or negative forms, or a mix of positive and negative items. The psychometric properties of the different versions (item analysis, dimensionality and reliability) were analyzed. Both negative and positive versions demonstrated better functioning than mixed versions. However, the mean total scores did not vary, which is an example of how similar means could mask other significant differences. Therefore, we advise against using mixed scales, and consider the use of positive or negative versions preferable.

€ Ver financiación

Información de financiación

Financiadores

Referencias bibliográficas

  • Amón Hortelano, J. (1990). Estadística para psicólogos: Estadística descriptiva [Statistics for psychologists: Descriptive statistics] (12th Ed.). Pirámide.
  • Arco-Tirado, J. L., Fernández-Martín, F. D., & Hoyle, R. H. (2018). Development and validation of a Spanish version of the Grit-S scale. Frontiers in Psychology, 9, Article 96. https://doi.org/10.3389/fpsyg.2018.00096
  • Areepattamannil, S., & Khine, M. S. (2018). Evaluating the psychometric properties of the original Grit Scale using rasch analysis in an Arab adolescent sample. Journal of Psychoeducational Assessment, 36(8), 856–862. http://doi.org/10.1177/0734282917719976
  • Carlson, M., Wilcox, R., Chou, C.-P., Chang, M., Yang, F., Blanchard, J., Marterella, A., Kuo, A., & Clark, F. (2011). Psychometric properties of reverse-scored items on the CES-D in a sample of ethnically diverse older adults. Psychological Assessment, 23(2), 558–562. https://doi.org/10.1037/a0022484
  • Chiavaroli, N. (2017). Negatively-worded multiple choice questions: An avoidable threat to validity. Practical Assessment, Research and Evaluation, 22, Article 3. https://doi.org/10.7275/ca7y-mm27
  • Cordero, A., Pamos, A., & Seisdedos, N. (2008). Inventario de Personalidad Neo Revisado (NEO PI-R), Inventario Neo Reducido de Cinco Factores (NEO-FFI): Manual profesional [The Neo Personality Inventory Revised (NEO-PI-R), the Reduced Five Factor Personality Inventory (NEO-PI-R): Professional manual] (3rd Ed.). TEA.
  • Costa, P. T., & McCrae, R. R. (1985). The NEO personality inventory manual. Psychological Assessment Resources.
  • Datu, J. A. D., King, R. B., Valdez, J. P. M., & Eala, M. S. M. (2019). Grit is associated with lower depression via meaning in life among Filipino high school students. Youth & Society, 51(6), 865–876. https://doi.org/10.1177/0044118x18760402
  • DiStefano, C., & Motl, R. W. (2009). Personality correlates of method effects due to negatively worded items on the Rosenberg Self-Esteem scale. Personality and Individual Differences, 46(3), 309–313. https://doi.org/10.1016/j.paid.2008.10.020
  • Duckworth, A. (2016). Grit: The power of passion and perseverance. Scribner/Simon & Schuster.
  • Duckworth, A. L., & Quinn, P. D. (2009). Development and validation of the Short Grit Scale (Grit–S). Journal of Personality Assessment, 91(2), 166–174. https://doi.org/10.1080/00223890802634290
  • Essau, C. A., Olaya, B., Anastassiou-Hadjicharalambous, X., Pauli, G., Gilvarry, C., Bray, D., O’Callaghan, J., & Ollendick, T. H. (2012). Psychometric properties of the Strength and Difficulties Questionnaire from five European countries. International Journal of Methods in Psychiatric Research, 21(3), 232–245. https://doi.org/10.1002/mpr.1364
  • Feldt, L. S. (1969). A test of the hypothesis that Cronbach’s alpha or Kuder-Richardson coefficient twenty is the same for two tests. Psychometrika, 34(3), 363–373. https://doi.org/10.1007/BF02289364
  • Ferrando, P. J., & Anguiano-Carrasco, C. (2010). El análisis factorial como técnica de investigación en psicología [Factor analysis as a research technique in psychology]. Papeles del Psicólogo, 31(1), 18–33.
  • Ferrando, P. J., & Lorenzo-Seva, U. (2017). Assessing the quality and appropriateness of factor solutions and factor score estimates in exploratory item factor analysis. Educational and Psychological Measurement, 78(5), 762–780. https://doi.org/10.1177/0013164417719308
  • García-Cueto, E. (1994). Coeficiente de congruencia [Congruence coefficient]. Psicothema, 6(3), 465–468.
  • Calderón Garrido, C., Navarro González, D., Lorenzo Seva, U., & Ferrando Piera, P. J. (2019). Multidimensional or essentially unidimensional? A multi-faceted factor-analytic approach for assessing the dimensionality of tests and items. Psicothema, 31(4), 450–457. https://doi.org/10.7334/psicothema2019.153
  • Gonzalez, O., Canning, J. R., Smyth, H., & MacKinnon, D. P. (2020). A psychometric evaluation of the Short Grit Scale. European Journal of Psychological Assessment, 36(4), 646–657. https://doi.org/10.1027/1015-5759/a000535
  • Haladyma, T. M., & Rodríguez, M. C. (2013). Developing and validating test items. Taylor & Francis. https://doi.org/10.4324/9780203850381
  • Horan, P. M., DiStefano, C., & Motl, R. W. (2003). Wording effects in self-esteem scales: Methodological artifact or response style? Structural Equation Modeling: A Multidisciplinary Journal, 10(3), 435–455. https://doi.org/10.1207/S15328007SEM1003_6
  • Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1–55. https://doi.org/10.1080/10705519909540118
  • Irwing, P., Booth, T., & Hughes, D. J. (2018). The Wiley handbook of psychometric testing: A multidisciplinary reference on survey, scale and test development. John Wiley & Sons. https://doi.org/10.1002/9781118489772
  • Józsa, K., & Morgan, G. A. (2017). Reversed items in likert scales: Filtering out invalid responders. Journal of Psychological and Educational Research, 25(1), 7–25.
  • Kaiser, H. F., & Rice, J. (1974). Little Jiffy, Mark Iv. Educational and Psychological Measurement, 34(1), 111–117. https://doi.org/10.1177/001316447403400115
  • Lane, S., Raymond, M. R., & Haladyma, T. M. (2016). Handbook of test development (2nd Rd.). Routledge. https://doi.org/10.4324/9780203102961
  • Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 22(140), 55.
  • Lloret-Segura, S., Ferreres-Traver, A., Hernández-Baeza, A., & Tomás-Marco, I. (2014). El análisis factorial exploratorio de los ítems: Una guía práctica, revisada y actualizada [Exploratory Item Factor Analysis: A practical guide revised and updated]. Anales de Psicología, 30(3). https://doi.org/10.6018/analesps.30.3.199361
  • Lorenzo-Seva, U., & Ferrando, P. J. (2013). FACTOR 9.2. Applied Psychological Measurement, 37(6), 497–498. https://doi.org/10.1177/0146621613487794
  • Marsh, H. W. (1986). Negative item bias in ratings scales for preadolescent children: A cognitive-developmental phenomenon. Developmental Psychology, 22(1), 37–49. https://doi.org/10.1037/0012-1649.22.1.37
  • Morell, M., Yang, J. S., Gladstone, J. R., Turci Faust, L., Ponnock, A. R., Lim, H. J., & Wigfield, A. (2021). Grit: The long and short of it. Journal of Educational Psychology, 113(5), 1038–1058. https://doi.org/10.1037/edu0000594
  • Moreno, R., Martínez, R. J., & Muñiz, J. (2015). Guidelines based on validity criteria for the development of multiple choice items. Psicothema, 27(4), 388–394. https://doi.org/10.7334/psicothema2015.110
  • Muñiz, J., & Fonseca-Pedrero, E. (2019). Diez pasos para la construcción de un test [Ten steps for test development]. Psicothema, 31(1), 7–16. https://doi.org/10.7334/psicothema2018.291
  • Musumari, P. M., Tangmunkongvorakul, A., Srithanaviboonchai, K., Techasrivichien, T., Suguimoto, S. P., Ono-Kihara, M., & Kihara, M. (2018). Grit is associated with lower level of depression and anxiety among university students in Chiang Mai, Thailand: A cross-sectional study. PLOS ONE, 13(12), Article e0209121. https://doi.org/10.1371/journal.pone.0209121
  • Navarro-González, D., Lorenzo-Seva, U., & Vigil-Colet, A. (2016). Efectos de los sesgos de respuesta en la estructura factorial de los autoinformes de personalidad [How response bias affects the factorial structure of personality self-reports]. Psicothema, 28(4), 465–470. https://doi.org/10.7334/psicothema2016.113
  • Ley Orgánica 3/2018, de 5 de diciembre, de Protección de Datos Personales y garantía de los derechos digitales [Organic Law 3/2019, of December 5, on the protection of personal data and guarantee of digital rights] (2018, December 6th). Boletín Oficial del Estado, 294, Sec. I, pp. 119788–119857. https://www.boe.es/eli/es/lo/2018/12/05/3/dof/spa/pdf
  • Paulhus, D. L., & Vazire, S. (2005). The self-report method. In Robins, R. W., Fraley, R. C., & Krueger, R. F. (Eds.), Handbook of research methods in personality psychology (pp. 224–239). Guildford Press.
  • Podsakoff, P. M., MacKenzie, S. B., & Podsakoff, N. P. (2012). Sources of method bias in social science research and recommendations on how to control it. Annual Review of Psychology, 63(1), 539–569. https://doi.org/10.1146/annurev-psych-120710-100452
  • Postigo, Á., Cuesta, M., García-Cueto, E., Menéndez-Aller, Á., González-Nuevo, C., & Muñiz, J. (2021). Grit assessment: Is one dimension enough? Journal of Personality Assessment, 103(6), 786–796. https://doi.org/10.1080/00223891.2020.1848853
  • Solís Salazar, M. (2015). The dilemma of combining positive and negative items in scales. Psicothema, 27(2), 192–199. https://doi.org/10.7334/psicothema2014.266
  • Suárez-Álvarez, J., Pedrosa, I., Lozano, L. M., García-Cueto, E., Cuesta, M., & Muñiz, J. (2018). Using reversed items in Likert scales: A questionable practice. Psicothema, 30(2), 149–158. https://doi.org/10.7334/psicothema2018.33
  • Timmerman, M. E., & Lorenzo-Seva, U. (2011). Dimensionality assessment of ordered polytomous items with parallel analysis. Psychological Methods, 16(2), 209–220. https://doi.org/10.1037/a0023353
  • van Sonderen, E., Sanderman, R., & Coyne, J. C. (2013). Ineffectiveness of reverse wording of questionnaire items: Let’s learn from cows in the rain. PLOS ONE, 8(7), Article e68967. https://doi.org/10.1371/journal.pone.0068967
  • Vazsonyi, A. T., Ksinan, A. J., Ksinan Jiskrova, G., Mikuška, J., Javakhishvili, M., & Cui, G. (2019). To grit or not to grit, that is the question! Journal of Research in Personality, 78, 215–226. https://doi.org/10.1016/j.jrp.2018.12.006
  • Vigil-Colet, A., Navarro-González, D., & Morales-Vives, F. (2020). To reverse or to not reverse likert-type items: That is the question. Psicothema, 32(1), 108–114. https://doi.org/10.7334/psicothema2019.286
  • Wong, N., Rindfleisch, A., & Burroughs, J. E. (2003). Do reverse-worded items confound measures in cross-cultural consumer research? The case of the material values scale. Journal of Consumer Research, 30(1), 72–91. https://doi.org/10.1086/374697
  • Woods, C. M. (2006). Careless responding to reverse-worded items: Implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment, 28(3), 186–191. https://doi.org/10.1007/s10862-005-9004-7
Fuente de los datos: Dialnet