Walk the Talk SoTL Contest Summary from 2nd Winning Team

Written by Maria Moore (COM), Cheri Simonds (COM), Lance Lippert (COM), Kevin Meyer (COM), Megan Koch (COM), and Derek Story (Director, HR Systems) at Illinois State University

On Wednesday, April 29, 2015, a reception was held to honor two award winning teams from our “Walk the Talk” contest for the best team or academic unit who applied SoTL research results/literature beyond the individual classroom to solve a problem, achieve a goal, or exploit an opportunity resulting in improved teaching or enhanced student learning at Illinois State University.

A summary of the first award winning project was featured in a recent blog post. Today’s blog highlights the work done by faculty in from the School of Communication in an effort to create a new teaching evaluation instrument.. This project, titled Creating a New Teaching Evaluation Instrument for the School of Communication is summarized below with an EXCELLENT list of SoTL literature readers interested in teaching evaluation can use, as appropriate:

What problem(s), goal(s) or opportunity(s) did your team seek to solve or exploit?

A teaching evaluation instrument is a critical strategic resource to ensure quality teaching within a division. It impacts practical issues such as promotion or raises, as well as being fundamental to assessing teaching quality.

We identified that our existing SoC instrument should be reviewed and updated. The assignment was given to our Teaching Effectiveness Committee who identified key problems with the old instrument, meeting bi-monthly revise the instrument. Members of the committee conducted a pilot test in their own courses at the end of the Fall 2013 semester, using both old and new instruments with students. The work of the committee, based on the successful pilot test, resulted in a recommendation for change to the full faculty body in March 2014. The faculty voted unanimously to accept the revision and the new instrument was implemented across all courses at the end of Spring 2014.

What strategies did your team devise to apply best practices from SoTL work to your problem/goal/opportunity?

We have many SoTL scholars in the SoC and it is a common practice to use SoTL literature as we identify problems or opportunities and devise resultant tactical strategies to solve or exploit them. This opportunity was no exception. At our first meeting we recognized the need to examine current literature and best practices to have external support and validation as a foundation for our recommendations.

We conducted a literature review on the current thinking about teaching evaluations as well as effective teaching practices. In addition to the literature reviewed, we also investigated teaching evaluation instruments from comparable and aspirational programs externally and internally.

What SoTL research (your own, colleagues, or from the literature) did you use to support your strategies?

We purposefully sought and reviewed literature from multiple disciplines, not just communication. We also sought current, as well as seminal scholarship on effective teaching practices. We included research from our own SoC scholars, too.

What were the outcomes and how were they assessed or measured?

Last year, one of our committee members ran factor analysis and scale reliabilities for all the SoC Spring 2014 data using the new teacher evaluation instrument. Results indicate that the new evaluation instrument we created performs very well in reliability tests, factor analysis, and predictive capability using regression procedures. And, it provides us the ability to condense aggregate reports of our evaluations into 4 categories or factors.

This analysis resulted in a spreadsheet template distributed to all faculty so they can also calculate their own aggregate factor scores for each subsequent semester. Faculty going for tenure or promotion use this spreadsheet as an element of their materials. Our SFSC also uses this spreadsheet to work with faculty deemed deficient as part of their performance enhancement strategic plan.

Please briefly reflect on the impact of this experience upon your team; in particular consider the specific role of the SoTL literature on your outcomes or consequences.

We accepted the responsibility to revise our evaluation instrument with great humility and a sense of tremendous responsibility. Simply put, we were entrusted with our colleagues’ future, as the evaluation instrument is one of the most important elements used to assess a teacher’s success or lack thereof. A foundation of SoTL literature (as we began and then navigated this responsibility) was both empowering and liberating. Empowerment occurred through the knowledge we gained about best practices as well as profoundly important SoTL research of both the student and the teacher perspectives of the evaluation process.

What are your team’s future plans for this particular project or initiative?

This is the first full academic year when all teachers in all roles in the SoC will use the new instrument. Our committee has the responsibility to continually assess its effectiveness for meeting our goals of the continual improvement of our teachers and our teaching. We are also experimenting with ways to mesh old data and new data from the old instrument and the new instrument for the multi-year reporting of data required for tenure and promotion applications.

What are your plans to make this work public?

We intend to submit a panel discussion about this topic for consideration at the next Central States Communication Association Meeting. We wanted to have this year’s aggregate data to discuss in addition to the project itself, so submission will occur in October 2015.

Literature Reviewed for Project

American Association of University Professors. (1990). Statement on Teaching Evaluations. Retrieved from http://www.aaup.org/report/statement-on-evaluation

Benton, S. L., Cashin, W. E., & Kansas, E. (2012). IDEA PAPER# 50 Student Ratings of Teaching: A Summary of Research and Literature.

Boysen, G. A. (2008). Revenge and student evaluations of teaching. Teaching of Psychology, 35(3), 218-222.

Calkins, S., & Micari, M. (2010). Less-than-perfect judges: Evaluating student evaluations. Thought & Action, 7.

Center for Research on Learning and Teaching University of Michigan. (n.d.). Gender and Student Evaluations: An Annotated Bibliography. Retrieved from http://www.crlt.umich.edu/sites/default/files/resource_files/gsebibliography.pdf

Chen, W., & Chen, W. (2010). Surprises learned from course evaluations. Research in Higher Education Journal, 9, 1-9.

Comadena, M., Hunt, S., & Simonds, C. (2007). The Effects of Teacher Clarity, Nonverbal Immediacy, and Caring on Student Motivation, Affective- and Cognitive Learning: A Research Note. Communication Research Reports24 (3), 241-248.

Cornell University Evaluation and Recognition of Teachers Handbook (n.d.). Retrieved from http://moodle.technion.ac.il/pluginfile.php/443177/mod_resource/content/1/Teaching%20Evaluation%20Handbook.pdf

Dodeen, H. (2013). College students’ evaluation of effective teaching: Developing an instrument and assessing its psychometric properties. Research in Higher Education Journal, 21, 1-12

DuCette, J., & Kenney, J. (1982). Do grading standards affect student evaluations of teaching? Some new evidence on an old question. Journal of Educational Psychology, 74(3), 308.

Feeley, H. T. (2002). Evidence of halo effects in student evaluations of communication instruction. Communication Education, 51(3), 225-236.

Frick, T. W., Chadha, R., Watson, C., Wang, Y., & Green, P. (2008, March). Theory-based course evaluation: Implications for improving student success in postsecondary education. In American Educational Research Association conference, New York.

Hudson, J. C. (1989). Expected Grades Correlate with Evaluation of Teaching.Journalism Educator, 44(2), 38-44.

Kim, C., Damewood, E., & Hodge, N. (2000). Professor attitude: Its effect on teaching evaluations. Journal of Management Education, 24(4), 458-473.

Kozey, S. R., & Feeley, H. T. (2009). Comparing Current and Former Student Evaluations of Course and Instructor Quality. Communication Research Reports, 26(2), 158-166.

Lewis, K. G. (2001). Making sense of student written comments. New Directions for Teaching and Learning, 2001(87), 25-32.

Marsh, H. W., & Roche, L. A. (1997). Making students’ evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. American Psychologist, 52(11), 1187.

Martin, E. (1984). Power and authority in the classroom: Sexist stereotypes in teaching evaluations. Signs, 482-492.

McCroskey, J. C. (1994). Assessment of affect toward communication and affect toward instruction in communication. In S. Morreale, & M. Brooks (Eds.),1994 SCA summer conference proceedings and prepared remarks: Assessing college student competence in speech communication. Annandale, VA: Speech Communication Association.

Onwuegbuzie, A. J., Witcher, A. E., Collins, K. M., Filer, J. D., Wiedmaier, C. D., & Moore, C. W. (2007). Students’ perceptions of characteristics of effective college teachers: A validity study of a teaching evaluation form using a mixed-methods analysis. American Educational Research Journal, 44(1), 113-160.

Ory, J. C. (2001). Faculty thoughts and concerns about student ratings. New directions for teaching and learning, 2001(87), 3-15.

Schrodt, P., Witt, P. L., Myers, S. A., Turman, P. D., Barton, M. H., & Jernberg, K. A. (2008). Learner empowerment and teacher evaluations as functions of teacher power use in the college classroom. Communication Education, 57(2), 180-200.

Sojka, J., Gupta, A. K., & Deeter-Schmelz, D. R. (2002). Student and faculty perceptions of student evaluations of teaching: A study of similarities and differences. College Teaching, 50(2), 44-49.

Wilson, R. C. (1986). Improving faculty teaching: Effective use of student evaluations and consultants. The Journal of Higher Education, 196-211.

Wines, W. A., & Lau, T. J. (2006). Observations on the folly of using student evaluations of college teaching for faculty evaluation, pay, and retention decisions and its implications for academic freedom. Wm. & Mary J. Women & L., 13, 167.

Wode, J., & Keiser, J. (2011). Online course evaluation literature review and findings. A report from Academic Affairs, Columbia College, Chicago.