Student Evaluations of Teaching Fail to Predict Learning: Meta-Analysis of Bias, Grade Inflation, and Incentive Distortion in Higher Education
DOI:
https://doi.org/10.65655/jscsaa20Keywords:
student evaluations of teaching, teaching effectiveness, grade inflation, higher education assessment, measurement bias, meta-analysis, incentive distortionAbstract
Background. End-of-course student evaluations of teaching (SETs) remain the dominant gauge of instructional quality, yet their validity and fairness have been repeatedly questioned. Purpose. This study re-examines whether SET scores capture durable learning and explores how high-stakes reliance on those scores reshapes academic behaviour. Methods. We integrated five complementary strands of secondary evidence: (a) a PRISMA-registered meta-analysis of 89 studies covering ≈5.4 million students, (b) re-analysis of two natural-experiment datasets with random instructor assignment, (c) psychometric audits of 14 institutional SET instruments, (d) computational text mining of 2.1 million open-ended comments, and (e) linkage of departmental SET means to alumni and employer outcomes. Results. Across studies, the pooled random-effects correlation between SETs and subsequent performance was r = 0.04 (95 % CI –0.03, 0.10), turning slightly negative after grade controls. Departments that tied contract renewal to minimum-SET thresholds exhibited a 0.27 GPA-point rise relative to matched controls, signalling grade inflation. Differential item functioning against female and racially minoritised faculty appeared in 9 of 23 common items, undermining measurement invariance. Programmes with high SET averages showed no advantage in alumni career readiness or employer satisfaction. Conclusions. Convergent evidence demonstrates that SETs fail to reflect long-term learning and introduce equity harms; their high-stakes use incentivises leniency that erodes academic standards. Universities seeking genuine teaching excellence should treat SETs as formative feedback, decouple them from punitive decisions, and adopt stakeholder-anchored, multi-measure frameworks that align evaluation with demonstrable learning.
References
ABET. (2025–2026). Criteria for accrediting engineering programs. https://www.abet.org/accreditation/accreditation-criteria/criteria-for-accrediting-engineering-programs-2025-2026/
Ambady, N., & Rosenthal, R. (1993). Half a minute: Predicting teacher evaluations from thin slices of nonverbal behavior and physical attractiveness. Journal of Personality and Social Psychology, 64(3), 431–441. https://doi.org/10.1037/0022-3514.64.3.431
American Sociological Association (ASA). (2019, February 13). Statement on student evaluations of teaching. https://www.asanet.org/wp-content/uploads/asa_statement_on_student_evaluations_of_teaching_feb132020.pdf
Banas, J. A., Dunbar, N., Rodriguez, D., & Liu, S.-J. (2011). A review of humor in educational settings: Four decades of research. Communication Education, 60(1), 115–144. https://doi.org/10.1080/03634523.2010.496867
Basow, S. A., & Martin, J. L. (2012). Bias in student evaluations. College Teaching, 60(1), 21-27. https://ldr.lafayette.edu/concern/publications/fb4948799
Boring, A., Ottoboni, K., & Stark, P. B. (2016). Student evaluations of teaching (mostly) do not measure teaching effectiveness. ScienceOpen Research, 1-11. https://doi.org/10.14293/S2199-1006.1.SOR-EDU.AETBZC.v1
Bjork, E. L., & Bjork, R. A. (2011). Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. In M. Gernsbacher, R. Pew, L. Hough, & J. Pomerantz (Eds.), Psychology and the real world (pp. 56-64). Worth. https://www.scirp.org/reference/referencespapers?referenceid=3200676
Braga, M., Paccagnella, M., & Pellizzari, M. (2014). Evaluating students' evaluations of professors. IZA Discussion Papers, No. 5620. https://www.iza.org/publications/dp/5620/evaluating-students-evaluations-of-professors
Bryant, J., Comisky, P. W., Crane, J. S., & Zillmann, D. (1980). Relationship between college teachers’ use of humor in the classroom and students’ evaluations of their teachers. Journal of Educational Psychology, 72(4), 511–519. https://doi.org/10.1037/0022-0663.72.4.511
Campbell, D. T. (1979). Assessing the impact of planned social change. Evaluation and Program Planning, 2(1), 67–90. https://doi.org/10.1016/0149-7189(79)90048-X
Carpenter, S. K., Wilford, M. M., Kornell, N., & Mullaney, K. M. (2013). Appearances can be deceiving: Instructor fluency increases perceptions of learning without increasing actual learning. Psychonomic Bulletin & Review, 20, 1350-1356. https://doi.org/10.3758/s13423-013-0442-z
Carrell, S. E., & West, J. E. (2010). Does professor quality matter? Evidence from random assignment of students to professors. Journal of Political Economy, 118(3), 409–432. https://doi.org/10.1086/653808
Cohen, P. A. (1981). Student ratings of instruction and student achievement: A meta-analysis of multisection validity studies. Review of Educational Research, 51(3), 281–309. https://doi.org/10.3102/00346543051003281
Dahabreh, I. J., Robertson, S. E., Petito, L. C., Hernán, M. A., & Steingrimsson, J. A. (2023). Efficient and robust methods for causally interpretable meta-analysis: Transporting inferences from multiple randomized trials to a target population. Biometrics, 79(2), 1057–1072. https://doi.org/10.1111/biom.13716
Deslauriers, L., McCarty, L. S., Miller, K., Callaghan, K., & Kestin, G. (2019). Measuring actual learning versus feeling of learning in response to being actively engaged in the classroom. Proceedings of the National Academy of Sciences, 116(39), 19251–19257. https://doi.org/10.1073/pnas.1821936116
Dik, B. J., & Duffy, R. D. (2009). Calling and vocation in career psychology: A pathway to purpose. Journal of Career Assessment, 17(3), 331-341. https://doi.org/10.1037/a0015547
Egger, M., Davey Smith, G., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315(7109), 629–634. https://doi.org/10.1136/bmj.315.7109.629
European Association for Quality Assurance in Higher Education. (2015). Standards and guidelines for quality assurance in the European Higher Education Area (ESG). https://ehea.info/media.ehea.info/file/2015_Yerevan/72/7/European_Standards_and_Guidelines_for_Quality_Assurance_in_the_EHEA_2015_MC_613727.pdf
Feldman, K.A. (1989). The association between student ratings of specific instructional dimensions and student achievement: Refining and extending the synthesis of data from multisection validity studies. Res High Educ 30, 583–645. https://doi.org/10.1007/BF00992392
Flaherty, C. (2019, September 9). Sociologists and more than a dozen other professional groups speak out against student evaluations of teaching. Inside Higher Ed. https://www.insidehighered.com/news/2019/09/10/sociologists-and-more-dozen-other-professional-groups-speak-out-against-student
Friedland S. (2025, September 11). ‘A for All’: Emory College faculty grapple with grade inflation. The Emory Wheel. Retrieved from https://www.emorywheel.com/article/2025/09/a-for-all-emory-college-faculty-grapple-with-grade-inflation
Gallup, Inc., & Purdue University. (2014). Great jobs, great lives: The Gallup-Purdue Index report. https://www.purdue.edu/uns/images/2014/gpi-alumnireport14.pdf
Geraghty, T. (2024, July 19). Goodhart’s law, Campbell’s law, and the Cobra Effect. Psych Safety. https://psychsafety.com/goodharts-law-campbells-law-and-the-cobra-effect/
Gilbert, R.O., Gilbert, D.R. (2025). Student evaluations of teaching do not reflect student learning: an observational study. BMC Med Educ 25, 313. https://doi.org/10.1186/s12909-025-06896-3
Goodhart, C. A. E. (1984). Problems of monetary management: The UK experience. In Monetary theory and practice: The UK experience (pp. 91–121). London: Palgrave Macmillan. https://doi.org/10.1007/978-1-349-17295-5
Gray, K. (2024, December 9). What are employers looking for when reviewing college students’ resumes? National Association of Colleges and Employers. https://www.naceweb.org/talent-acquisition/candidate-selection/what-are-employers-looking-for-when-reviewing-college-students-resumes
Hartung, J. (1999). An alternative method for meta-analysis. Biometrical Journal, 41(8), 901–916. https://doi.org/10.1002/(SICI)1521-4036(199912)41:8
Hirsch, A. (2025, March 27). What if student evaluations measured what actually matters? Center for Innovative Teaching and Learning. Retrieved from https://citl.news.niu.edu/2025/03/27/what-if-student-evaluations-measured-what-actually-matters
Holmström, B., & Milgrom, P. (1991). Multitask principal-agent analyses: Incentive contracts, asset ownership, and job design. Journal of Law, Economics, & Organization, 7(1), 24-52. Retrieved from https://people.duke.edu/~qc2/BA532/1991%20JLEO%20Holmstrom%20Milgrom.pdf
Huemer, M. (2001). Student evaluations: A critical review. Unpublished manuscript. https://spot.colorado.edu/~huemer/papers/sef.htm
Indiana University Center for Postsecondary Research. (2024). NSSE 2024 annual results: Engagement insights. National Survey of Student Engagement. https://nsse.indiana.edu/nsse/reports-data/nsse-overview.html
IntHout, J., Ioannidis, J. P. A., Rovers, M. M., & Goeman, J. J. (2016). Plea for routinely presenting prediction intervals in meta-analysis. BMJ Open, 6(7), e010247. https://doi.org/10.1136/bmjopen-2015-010247
Knapp, G., & Hartung, J. (2003). Improved tests for a random-effects meta-regression with a single covariate. Statistics in Medicine, 22(17), 2693-2710. https://doi.org/10.1002/sim.1482
Kogan, V., Genetin, B., Chen, J., & Kalish, A. (2022, January 5). Students' grade satisfaction influences evaluations of teaching: Evidence from individual-level data and an experimental intervention (EdWorkingPaper No. 22-513). Annenberg Institute at Brown University. https://doi.org/10.26300/spsf-tc23
Kornell, N. (2013, May 31). Do the best professors get the worst ratings? Psychology Today. https://www.psychologytoday.com/us/blog/everybody-is-stupid-except-you/201305/do-the-best-professors-get-the-worst-ratings
Knapp, G., & Hartung, J. (2003). Improved tests for a random effects meta-regression with a single covariate. Statistics in Medicine, 22(17), 2693–2710. https://doi.org/10.1002/sim.1482
MacNell, L., Driscoll, A., & Hunt, A. N. (2015). What’s in a name: Exposing gender bias in student ratings of teaching. Innovative Higher Education, 40(4), 291–303. https://doi.org/10.1007/s10755-014-9313-4
Marsh, H. W., & Roche, L. A. (1997). Making students’ evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. American Psychologist, 52(11), 1187–1197. https://doi.org/10.1037/0003-066X.52.11.1187
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741
Naftulin, D. H., Ware, J. E., Jr., & Donnelly, F. A. (1973). The Doctor Fox lecture: A paradigm of educational seduction. Journal of Medical Education, 48(7), 630–635. https://doi.org/10.1097/00001888-197307000-00003
National Association of Colleges and Employers. (2025a, January). Job Outlook 2025. https://www.naceweb.org/docs/default-source/default-document-library/2025/publication/research-report/2025-nace-job-outlook-jan-2025.pdf
National Association of Colleges and Employers. (2025b, January 13). The gap in perceptions of new grads’ competency proficiency and resources to shrink it. https://www.naceweb.org/career-readiness/competencies/the-gap-in-perceptions-of-new-grads-competency-proficiency-and-resources-to-shrink-it
Orwin, R. G. (1983). A fail-safe N for effect size in meta-analysis. Journal of Educational Statistics, 8(2), 157-159. https://doi.org/10.3102/10769986008002157
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., … & Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, n71. https://doi.org/10.1136/bmj.n71
Radavoi, C.N., Quadrelli, C. & Collins, P. (2025). Moral Responsibility for Grade Inflation: Where Does It Lie?. J Acad Ethics. https://doi.org/10.1007/s10805-025-09627-x
Sacks, P. (1996). Generation X goes to college: An eye-opening account of teaching in post-modern America. Open Court. https://www.petersacks.org/generation_x_goes_to_college__an_eye_opening_account_of_teaching_in_postmodern_am_2221.htm
Sangwa, S., & Mutabazi, P. (2025). Mission-Driven Learning Theory: Ordering Knowledge and Competence to Life Mission. Preprints. https://doi.org/10.20944/preprints202509.0906.v1
Sixbert S., Titus L., Simeon N., Placide M. (2025). Expertise-Autonomy Equilibria in African Higher Education: A Systematic Review of Student-Centred Pedagogies and Graduate Readiness. International Journal of Research and Innovation in Social Science (IJRISS), 9(03), 5419-5432. DOI: https://dx.doi.org/10.47772/IJRISS.2025.903SEDU0394
Sparks, S. D. (2011, April 26). Studies find 'desirable difficulties' help students learn. Education Week. https://www.edweek.org/leadership/studies-find-desirable-difficulties-help-students-learn/2011/04
Stark, P. B., & Freishtat, R. (2014). An evaluation of course evaluations. ScienceOpen Research. https://www.scienceopen.com/hosted-document?doi=10.14293/S2199-1006.1.SOR-EDU.AOFRQA.v1
Sterne, J. A. C., & Egger, M. (2001). Funnel plots for detecting bias in meta-analysis: Guidelines on choice of axis. Journal of Clinical Epidemiology, 54(10), 1046-1055. https://doi.org/10.1016/S0895-4356(01)00377-8
Uttl, B., White, C. A., & Gonzalez, D. W. (2017). Meta-analysis of faculty's teaching effectiveness: Student evaluation of teaching ratings and student learning are not related. Studies in Educational Evaluation, 54, 22–42. https://doi.org/10.1016/j.stueduc.2016.08.007
Zumbo, B.D. (1999). A Handbook on the Theory and Methods of Differential Item Functioning (DIF) LOGISTIC REGRESSION MODELING AS A UNITARY FRAMEWORK FOR BINARY AND LIKERT-TYPE (ORDINAL) ITEM SCORES. https://www.semanticscholar.org/paper/A-Handbook-on-the-Theory-and-Methods-of-Item-(DIF)-Zumbo/7f88fb0ad98645582665532600d7c46406fa2db6
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Sixbert SANGWA, Simeon Nsabiyumva (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
