The SoTL Advocate

Supporting efforts to make public the reflection and study of teaching and learning at Illinois State University and beyond…

Common Practical Problems in SoTL Research Design

1 Comment

Written by Jennifer Friberg, SoTL Scholar-Mentor at Illinois State University

Recently, I was reading through a book titled Doing the Scholarship of Teaching and Learning: Measuring Systemic Changes to Teaching and Improvements in Learning (Regan A. R. Gurung and Janie H. Wilson, editors). In this text, Robert Bartsch authored a chapter focused on the practicality of designing SoTL studies and suggested solutions to common practical problems encountered by novice and veteran SoTL researchers. I’ve summarized Bartsch’s advice to SoTL researchers below:

  1. I Have to Measure Everything. It is not unusual for SoTL researchers to be ambitious, designing SoTL studies where masses of data are collected, then analyzed. Practically speaking, there are times when having large quantities of data might not be helpful. Rather, collecting excesses of data can complicate data analysis and yield less useful information than a carefully focused study of  one learning context or stakeholder as a data source. If you need many data points, perhaps consider multiple studies to remain focused on high-quality work. Don’t feel like you need to measure everything at once!
  2. I Do Not Have Many Students. In some disciplines, small class sizes can limit student participant numbers in SoTL research projects which can impact the statistical viability of a SoTL study. Bartsch suggests that selecting the right research design can mitigate issues with sample size. He suggests improving statistical power by using a within participants design such as a pre-test/post-test method. If statistical analysis is difficult secondary to low participant group numbers, consider the use of qualitative measures to study student learning via portfolio/artifact review or student perceptions/reflections. Don’t feel limited by a small participant group!
  3. I Only Have a Single Class. Some SoTL research designs can be used with a single group (or class) of students. However, some researchers might prefer to subdivide a group of students to allow for the comparison of control vs. treatment conditions within a class. Bartsch suggests that there are ways of splitting a single class into multiple groups to address this issue in an ethical manner (e.g., splitting a single class into sections, using different classrooms for each group of students, administration of treatment and control conditions simultaneously within the same classroom context). Use qualitative approaches such as focus groups or interviews to ascertain changes in learning over time. Don’t let a small participant group impact your ability to engage in SoTL!
  4. Random Assignment Sounds Great, But How Can I Do It? Though not all SoTL studies require random participant groups, some will. And, a truly randomized sample is difficult to come by, even if you teach two sections of the same course as there are many situational factors that impact decisions regarding how students register for a class (e.g., day/time of class, students wanting to register with friends, etc.). If you aren’t able to randomize enrollment in class sections or groups in a single class, that is merely a limitation to your SoTL study. Report it, and continue your study. Don’t let a lack of randomization keep you from SoTL research!
  5. I Want to Detect a Subtle Effect. Be realistic in the design of your SoTL study. If you are seeking to research the impact of a teaching support/assignment/approach that is only used once or twice in the semester, it might be difficult to statistically measure learning outcomes. Rather, make a choice to measure larger effects (variables applied often to measure more detectable changes) statistically or choose qualitative methods to gather data to reflect smaller changes in a short amount of time. Don’t miss the opportunity to study incremental student learning!

Are these issues you’ve had as a SoTL researcher? How have you overcome these issues? What was the best approach for you and for your students? We’d love to hear about your experiences in the comments section below.

Blog References:

Gurung, R. A., R. & Wilson, J. H. (Eds.). (2013). Doing the scholarship of teaching and learning: Measuring systematic changes to teaching and improvements in learning. New Directions for Teaching and Learning, Number 136, Winter 2013.


One thought on “Common Practical Problems in SoTL Research Design

  1. Pingback: Taking a Winter Break | The SoTL Advocate

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s