Development and validation of the CIS rubric for business simulator implementation
Keywords:
business simulations; implementation fidelity; rubric; content validity; inter‑rater reliability.Abstract
Objective. To present the development, content validity, and inter‑rater reliability of the CIS rubric to assess implementation quality in business simulation courses in higher education. Method. Instrumental study; CIS has three criteria (Content support; Techno‑functional management; Development & debriefing) and four levels (1–4). Content validity via expert judgement (CVR, I‑CVI, S‑CVI/Ave). Inter‑rater reliability via ICC(2,k) with 95% CIs. Results: S‑CVI/Ave=.96; total ICC(2,k)=.92 (95% CI [.86, .98]); criteria .84–.90. Conclusions. CIS shows solid content validity and inter‑rater reliability and can be adopted for instructional monitoring and continuous improvement.
Downloads
References
AERA, APA, & NCME. (2014). Standards for educational and psychological testing. American Educational Research Association.
Alvarez‑Aranzamendi, H. (2021). Uso de simuladores en una escuela de negocios como herramienta de aprendizaje de habilidades gerenciales para estudiantes de pregrado en administración. Company Games & Business Simulation Academic Journal, 1(1), 49–58. https://uajournals.com/ojs/index.php/businesssimulationjournal/article/view/868
Alvarez Aranzamendi, H. A., & Alvarez Aranzamendi, C. L. (2022). Desarrollo de competencias digitales de estudiantes universitarios en el contexto de uso de simuladores de negocio.
Company Games & Business Simulation Academic Journal, 2(2), 31–36. https://uajournals.com/ojs/index.php/businesssimulationjournal/article/view/1283
Alvarez Aranzamendi, H. (2023). Los simuladores de negocios y la inteligencia artificial generativa aplicada en la educación universitaria: Situación del estado del arte. Company Games & Business Simulation Academic Journal, 3(2), 87–92. https://www.uajournals.com/ojs/index.php/businesssimulationjournal/article/view/1512
Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3–4), 327–350. https://doi.org/10.1007/s10464- 008-9165-0
Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17–64). Praeger.
Kolb, D. A. (1984). Experiential learning: Experience as the source of learning and development. Prentice‑Hall.
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012
Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28(4), 563–575. https://doi.org/10.1111/j.1744-6570.1975.tb01393.x
Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–386. https://doi.org/10.1097/00006199-198611000-00017
O’Donnell, C. L. (2008). Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K–12 curriculum intervention research. Review of Educational Research, 78(1), 33–84. https://doi.org/10.3102/0034654307313793
Polit, D. F., & Beck, C. T. (2006). The content validity index: Are you sure you know what's being reported? Research in Nursing & Health, 29(5), 489–497. https://doi.org/10.1002/nur.20147
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. https://doi.org/10.1037/0033-2909.86.2.420
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Hugo Alvarez-Aranzamendi
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
.jpg)
