The SoTL Advocate

Supporting efforts to make public the reflection and study of teaching and learning at Illinois State University and beyond…


Leave a comment

Measuring Student Learning in SoTL Project

Written by Kathleen McKinney, Cross Endowed Chair in SoTL, Illinois State University

In much of our SoTL work/projects we need and want to measure some type(s) of student learning and other outcomes. When considering how to do this, start by thinking about and, perhaps, discussing with others the following more general but related questions. What, specifically, are the learning outcomes that you think students should be demonstrating? What might be the intervening processes (the why and how) between the teaching or intervention or material…and the learning outcomes? What does past research tell you about what learning and other outcomes to measure and how? What do theories about learning (general and specific to your topic) tell you about what outcomes to measure and how? Given your SoTL research questions and ways you hope to use your data, do you need measures that allow statistical analysis and generalization and/or data that gives student ‘voices’ and offers description?

Next, think about the different types of outcomes you might need to measure including the following:

  • Student attitudes/perceptions/beliefs/values.
  • Student affective development.
  • Student skills.
  • Student content knowledge.
  • Student use/application/transfer of knowledge or skills.
  • Student retention of knowledge, skills, or affective change over time.
  • Student communication and interactions with peers, faculty/staff.

We can also think about measuring outcomes in terms of several measurement dimensions, each of which can be seen as dichotomies or as continuums. First, measures of student outcomes may be more or less indirect (e.g. student perceptions), direct (e.g., valid instruments/measures of learning), and/or performance (e.g., behaviors). Second, measures could be static at one point in time or growth/change measures (pre-post or follow-up measures after some period of time). Third, we may use single or multiple measures for a given student outcome. Fourth, sometimes we utilize existing instruments; for other situations, creating your own instruments may be more appropriate. Fifth, measures may be quantitative and/or qualitative.

Finally, the ways we measure student learning and other outcomes in our SoTL projects occur within a broader methodology or study design or general data gathering strategy. For example, we may analyze student products, conduct interviews or focus groups with students, have them complete questionnaires, give them pre-post tests, ask them to write learning reflection essays or journals, facilitate and record ‘think-alouds’ of an appropriate learning task, conduct a quasi-experiment, use observational research, and more. Be on the look out for future posts to this blog by Dr. Jennifer Friberg with details on several of these SoTL research methods/strategies.


1 Comment

SoTL Methods Series #2: Content Analysis

Written by Jennifer Friberg, CSD/SoTL Scholar-Mentor at Illinois State University

Content analysis is defined as systematic and replicable analysis of written, spoken or visual data in order to make inferences from that data with regard to a particular context (Friberg & Cox, 2014). Fraenkel and Wallen describe content analysis as follows (2003, p. 482):

“Content analysis as a methodology is often used in conjunction with other meds, in particular historical and ethnographic research. It can be used in any context in which [a] researcher desires a means of systemizing and (often) quantifying data. It is extremely valuable in analyzing observation and interview data.”

Content analysis can be conducted in an inductive or deductive manner. Both inductive and deductive content analyses involve the arrangement of data into categories for interpretation; however, there are differences in how this occurs. Bishop-Clark and Dietz-Uhler (2012) describe the types of content analysis as follows:

  • Inductive content analysis occurs by sifting through data to identify themes that emerge organically from the data being analyzed.
  • Deductive content analysis occurs when researchers utilize categories established in earlier research, often in an effort to formulate, support, or refute a theory.

The determination of whether to use inductive or deductive content analysis is typically guided by the topic being studied. Data analyzed through content analysis is coded for interpretation. Content analysis is often used in conjunction with other research methods as part of a mixed-method project.

Exemplar articles using a case study methodology include the following:

Gelbman, S. M. (2011). A qualitative assessment of the learning outcomes of teaching introductory American politics in comparative perspective. Journal of Political Science Education, 7(4), 359–374.

Maldoni, A., Kennelly, R. & Davies, D. (2009). Integrating discipline-based reading to improve intercultural and international learning. International Journal for the Scholarship of teaching and Learning, 3(1), article 8.

Blog References:

Bishop-Clark, C. & Dietz-Uhler, B. (2012). Engaging in the scholarship of teaching and learning: A guide to the process, and how to develop a project from start to finish. Sterling, VA: Stylus Publishing.

Fraenkel, J. R. & Wallen, N. E. (2003). How to design and evaluate research in education (5th ed). New York: McGraw-Hill.

Friberg, J. & Cox, M. (2014, October). Selecting methodologies for your SoTL research projects workshop: Supplemental workshop resource. Unpublished paper.


Leave a comment

More on SoTL ‘Stories’: Motivations for and Roles in SoTL

Written by Kathleen McKinney, Cross Endowed Chair in SoTL, Illinois State University

In a recent post on this blog, Jennifer Friberg, SoTL Scholar-Mentor at Illinois State University, shared a bit of her SoTL story –why she does SoTL, how she came to be involved in SoTL, how her SoTL career developed over time. Jennifer was responding to a blog post by Janice Miller-Young, director of the Institution for Scholarship of Teaching and Learning at Mt. Royal University titled “How to tell the story of SoTL.” In her post, Janice suggested that faculty engage in SoTL research for three main purposes (quoting Jennifer): “to generate and study innovations in teaching, to apply and study innovative pedagogies, and to better understand the complexities in teaching and learning.” In this post, I add to these SoTL stories by summarizing highlights from my SoTL story then, based on this story as well as observations of, and conversations with, others in the field, I list 1. motivations for doing SoTL and 2. SoTL roles.

Thirty years ago, in 1985, I published with a colleague about our sociology curriculum and advising practices. Though our ideas were based in past literature, the work was in the ‘we tried it, we liked it’ anecdotal genre. My SoTL work stayed discipline-specific and somewhat ‘tips’ oriented as I shared what I believed were best teaching practices. I had the privilege of meeting excellent role models and mentors in the teaching-learning movement of the American Sociological Association beyond my institution. Over time, my SoTL work became focused on questions that arose from my lived experiences helping students learn inside and outside the classroom, and was evidence-based and peer-reviewed. In the mid 1990s I had the opportunity to serve as Editor of Teaching Sociology and learning so much from every submission I had the joy of reading. As I moved into an institutional role as director of our teaching center, I began to do more and ‘better’ SoTL work on sociology student learning and to help others do SoTL. I had the chance to work with amazing folks as a Carnegie SoTL Scholar, meeting more wonderful people from many disciplines and around the globe. I became an Endowed Chair in SoTL at my institution, began to write about the field of SoTL, and increased my involvement in SoTL research and service in my discipline and in the international, cross-discipline SoTL field through ISSOTL among other organizations and contexts. My focus now, as a (mostly) retired faculty member, is to provide service, mentoring, and support to others doing SoTL. I also continue to learn from others doing SoTL and writing about SoTL.

I offer nine forms of motivations as to why people do the scholarship of teaching and learning.

  • It builds our vitae and our cases for rewards and promotions.
  • It is valued by others in our institution and/or discipline.
  • To improve our teaching.
  • To improve our students’ learning outcomes.
  • To involve students as co-researchers in SoTL–a high impact teaching-learning practice.
  • To help our department, discipline or institution with high priority initiatives, assessment, accreditation, and program review.
  • Because we cherish our interactions with other SoTLers.
  • Because it becomes part of our professional, and even personal, identity.
  • Because we enjoy the specifics of the ‘work’ itself.

I think there are several types of roles in which we choose to engage within the field of SoTL. Of course, many of us engage in more than one of these roles at the same time and/or over the course of our SoTL careers. These roles include consuming SoTL (read, listen to, adapt, use others’ SoTL work); producing SoTL (conduct and make public original SoTL work); being an active colleague in the SoTL community in your institution and beyond and in your discipline and beyond (e.g., attend conferences, join organizations, use SoTL beyond your classroom…); and/or serving the SoTL field (e.g., editor, mentor, committee member in a SoTL professional organization…).

I look forward to hearing some other SoTL ‘stories’, including examples of these or other motivations for and roles in SoTL.


2 Comments

SoTL Methods Series #1: Case Study Research

Written by Jennifer Friberg, CSD/SoTL Scholar-Mentor at Illinois State University

Case study research is any inquiry which involves careful description and analysis of a specific individual, group or event. Bishop Clark and Dietz-Ulher (2012, p. 50) explain:

“While many SoTL projects involve hundreds of students, the work involved in a case study has a sample size, or n, of one – whether it be one classroom or one student. However, the work involved in something as simple as an investigation of a single student processing a single assignment can be substantial and complex.”

Why might a researcher decide to select a case study methodology for a research study? Case studies can provide rich, descriptive data of a single case, yielding understating of a phenomenon more deeply (Patton, 2002). Case studies are appropriate to adopt as a research method in conditions where a researcher seeks to fully and deeply understand the complexities of a single phenomenon. Researchers should be cautioned, however, that due to a case study’s intense focus on a single source of data, results from case study research might not be easily generalizable to a larger population.

Soy (1997) suggested several suggestions for conducting case-study research. These have been adapted below to focus specifically on the design of case study-based SoTL research, but readers can access the original source here for the complete reference. Important considerations in the design of case study research are as follows:

  1. Determine your case to study and define your research question. To engage in SoTL research using a case study methodology, the researcher must first determine the entity to be studied. Perhaps this is a single assignment or one classroom of students. It could be a single student or a single teacher. Any single agent involved in the teaching and learning process can be studied as a discreet “case” in some fashion. The focus of the case study is then studied in depth in light of research questions posed by the researcher. These questions can be specific or general in nature, depending on the purpose and audience for your study.
  2. Determine data gathering and analysis techniques. Depending on the research questions to be answered over the course of a particular SoTL study, case study data can be collected in a variety of ways (i.e., interviews, observations, reflections). Data should be collected after advanced planning of how data will be analyzed to ensure that data is collected to appropriately and adequately address various research questions. Analysis of data can be qualitative or quantitative, depending on researcher preferences.

Researchers pondering the use of a case study methodology for a SoTL project can refer to these exemplar articles to better understand how case study research is conducted and reported:

Moore, M. A., & Bruckner, I. M. (2010). A case of collaboration: Faculty experiences within a multidisciplinary, multimedia, multi-campus learning community in an urban community college district. MountainRise, 6(2). http://mountainrise.wcu.edu/index.php/MtnRise/article/view/135

Weller, S., Domarkaite, G. K., Lam, J. L. C., & Metta, L. U. (2013). Student-faculty co-inquiry into student reading: Recognizing SoTL as pedagogic practice. International Journal for the Scholarship of teaching and Learning, 7(2), article 9.

Finally, the following non-research reference might be helpful to any scholars seeking more information about case study-based research:

Yin, R. K. (2008). Case study research: Design and methods. Newbury Park, CA: Sage.

Blog References:

Bishop-Clark, C. & Dietz-Uhler, B. (2012). Engaging in the scholarship of teaching and learning: A guide to the process, and how to develop a project from start to finish. Sterling, VA: Stylus Publishing.

Friberg, J. & Cox, M. (2014, October). Selecting methodologies for your SoTL research projects workshop: Supplemental workshop resource. Unpublished paper.

Patton, M. Q. (2002). Qualitative research & evaluation methods (3rd ed.). Thousand Oaks, CA: Sage Publications.

Soy, S. (1997). The case study as a research method. Retrieved from https://www.ischool.utexas.edu/~ssoy/usesusers/l391d1b.htm


1 Comment

SoTL Research Methodology Series

Over the next several weeks, six blog entries will be posted to discuss research methodologies common to SoTL research. These posts were inspired by a workshop given for Illinois State faculty and students in the fall of 2014. As we chatted with personnel from across campus about methodologies common in SoTL research, it became evident that faculty from different disciplines might vary as to the definition and preferences for implementation of the research methods identified for discussion in this SoTL Methods Series. We honor and celebrate those differences and offer the following caveat: the content of this blog series is meant to be a general introduction to research methods for faculty or students who might be interested in learning about common SoTL research methods. We would also offer that any of the research methods presented could be used in conjunction with one another to triangulate data and craft a mixed methods study.

As always, ISU faculty and students are encouraged to contact the office of the Cross Chair in SoTL at ISU (sotl@ilstu.edu) with questions or to schedule a consultation to discuss or brainstorm SoTL research projects.


Leave a comment

WALK THE TALK: A Scholarship of Teaching and Learning Contest for best application of SoTL knowledge beyond the individual classroom

Written by Kathleen McKinney, Cross Endowed Chair in the Scholarship of Teaching and Learning, Illinois State University

SoTL has a number of purposes or types of value. Most important is its impact on teaching and student learning via application of SoTL data and results. I have been interested, for many years, in the application/use of SoTL work at the classroom level but, also, beyond…at the program, department, institutional, and disciplinary levels. I have published essays on the topic, given keynote presentations about this issue, worked in my disciplinary organization to encourage and reward such application, served for several years on the International Society for the Scholarship of Teaching and Learning’s Advocacy and Outreach Committee, prepared and discussed ideas with our Provost about using SoTL within the institutional culture, and created faculty development events and opportunities to encourage application. This interest and all of these experiences led to my most recent idea—a contest to recognize the best example of an application of SoTL beyond the classroom at our institution.

I knew I needed help so I turned the idea over to Department of Communication faculty member and past ISU SoTL Scholar-Mentor, Dr. Maria Moore. She took the lead on naming the contest, creating the Call for Applications and the application questions, developing public relations materials, and (later this spring) planning the public recognition event. In the spirit of good teaching-learning practice and SoTL, Maria involved a Public Relations student, Tyler Eilts, and a Graphic Design student, Kelsy Brewer, to assist her.

Though this contest is limited to teams at Illinois State University, I thought those from other institutions might find the idea of interest. The purpose of the ‘Walk the Talk’ contest is “to recognize and encourage systematic and specific application of best practices discovered through SoTL research/literature to the teaching and learning of ISU students beyond an individual classroom.” And “is intended to recognize the best team or academic unit that applied SoTL research beyond the individual classroom to solve a problem, achieve a goal, or exploit an opportunity resulting in improved teaching or enhanced student learning.” Interdisciplinary teams or academic units are also welcome to apply.

Given the goals of the contest and to provide sufficient detail for selection of award recipients, as well as to have consistent information across applications, we decided to ask teams to complete a series of specific questions (http://sotl.illinoisstate.edu/downloads/WalkTheTalk2015App.pdf). These application questions then align with our selection criteria:

  • Clear description of significant application/impact of SoTL work on teaching and student learning or development to solve a problem, achieve a goal, or exploit an opportunity. The SoTL work may be work by the members of the team and/or SoTL from existing literature.
  • Concrete and appropriate evidence linking successful student attitudinal, learning or developmental outcomes to the specific SoTL work/literature.
  • Impact of the application of SoTL work is beyond the individual classroom level.

The selected award recipients (first place and honorable mention) will be recognized in a variety of ways including monetary awards, a plaque, a public celebration event, and visibility in University websites and/or publications. Recipients, however, will also be expected to help us share their application success story by writing a brief report or reconfiguring their application materials for our SoTL at ISU newsletter and The SoTL Advocate Blog.

For the complete call for applications, go to http://sotl.illinoisstate.edu/walkTheTalk.shtml


1 Comment

Why do you SoTL?

Written by Jennifer Friberg, SoTL Scholar-Mentor at Illinois State University

Dr. Janice Miller-Young, director of the Institution for Scholarship of Teaching and Learning at Mt. Royal University, recently authored a blog post title “How to tell the story of SoTL,” focusing on the reasons that researchers engage in SoTL inquiry at her university. In this post, she suggests that faculty engage in SoTL research for three main purposes: to generate and study innovations in teaching, to apply and study innovative pedagogies, and to better understand the complexities in teaching and learning. Likely most SoTL researchers fall into at least one of those categories. .

My involvement in SoTL began with my desire to understand my students more thoroughly as learners. I sought evidence to connect what I subjectively observed as an instructor to the actual reality for my students. I wanted to know what activities, experiences, and methods were most successful in helping them to learn and expand upon course material. My initial SoTL studies helped me to move beyond scholarly teaching to become a scholar of teaching and learning. And, while I still engage in such studies, my interests have turned to studies which might help build the commons (Huber and Hutchings, 2005), synthesizing and combining my SoTL work with that of other professionals to allow for broader understanding of teaching and learning questions. In short, my SoTL work crosses all three categories suggested by Dr. Miller-Young in her post, though I would add that I also engage in SoTL to give my students an active voice in their learning processes.

Why do you SoTL? What motivates you to engage in SoTL and share it with others? Feel free to comment below!

Huber, M. T. and Hutchings, P. (2005). The Advancement of Learning: Building the Teaching Commons. San Francisco: Jossey-Bass.