The SoTL Advocate

Supporting efforts to make public the reflection and study of teaching and learning at Illinois State University and beyond…

1 Comment

Common Practical Problems in SoTL Research Design

Written by Jennifer Friberg, SoTL Scholar-Mentor at Illinois State University

Recently, I was reading through a book titled Doing the Scholarship of Teaching and Learning: Measuring Systemic Changes to Teaching and Improvements in Learning (Regan A. R. Gurung and Janie H. Wilson, editors). In this text, Robert Bartsch authored a chapter focused on the practicality of designing SoTL studies and suggested solutions to common practical problems encountered by novice and veteran SoTL researchers. I’ve summarized Bartsch’s advice to SoTL researchers below:

  1. I Have to Measure Everything. It is not unusual for SoTL researchers to be ambitious, designing SoTL studies where masses of data are collected, then analyzed. Practically speaking, there are times when having large quantities of data might not be helpful. Rather, collecting excesses of data can complicate data analysis and yield less useful information than a carefully focused study of  one learning context or stakeholder as a data source. If you need many data points, perhaps consider multiple studies to remain focused on high-quality work. Don’t feel like you need to measure everything at once!
  2. I Do Not Have Many Students. In some disciplines, small class sizes can limit student participant numbers in SoTL research projects which can impact the statistical viability of a SoTL study. Bartsch suggests that selecting the right research design can mitigate issues with sample size. He suggests improving statistical power by using a within participants design such as a pre-test/post-test method. If statistical analysis is difficult secondary to low participant group numbers, consider the use of qualitative measures to study student learning via portfolio/artifact review or student perceptions/reflections. Don’t feel limited by a small participant group!
  3. I Only Have a Single Class. Some SoTL research designs can be used with a single group (or class) of students. However, some researchers might prefer to subdivide a group of students to allow for the comparison of control vs. treatment conditions within a class. Bartsch suggests that there are ways of splitting a single class into multiple groups to address this issue in an ethical manner (e.g., splitting a single class into sections, using different classrooms for each group of students, administration of treatment and control conditions simultaneously within the same classroom context). Use qualitative approaches such as focus groups or interviews to ascertain changes in learning over time. Don’t let a small participant group impact your ability to engage in SoTL!
  4. Random Assignment Sounds Great, But How Can I Do It? Though not all SoTL studies require random participant groups, some will. And, a truly randomized sample is difficult to come by, even if you teach two sections of the same course as there are many situational factors that impact decisions regarding how students register for a class (e.g., day/time of class, students wanting to register with friends, etc.). If you aren’t able to randomize enrollment in class sections or groups in a single class, that is merely a limitation to your SoTL study. Report it, and continue your study. Don’t let a lack of randomization keep you from SoTL research!
  5. I Want to Detect a Subtle Effect. Be realistic in the design of your SoTL study. If you are seeking to research the impact of a teaching support/assignment/approach that is only used once or twice in the semester, it might be difficult to statistically measure learning outcomes. Rather, make a choice to measure larger effects (variables applied often to measure more detectable changes) statistically or choose qualitative methods to gather data to reflect smaller changes in a short amount of time. Don’t miss the opportunity to study incremental student learning!

Are these issues you’ve had as a SoTL researcher? How have you overcome these issues? What was the best approach for you and for your students? We’d love to hear about your experiences in the comments section below.

Blog References:

Gurung, R. A., R. & Wilson, J. H. (Eds.). (2013). Doing the scholarship of teaching and learning: Measuring systematic changes to teaching and improvements in learning. New Directions for Teaching and Learning, Number 136, Winter 2013.


Leave a comment

Advice for New SoTL Researchers

Written by Jennifer Friberg, SoTL Scholar-Mentor at Illinois State University

Perhaps it’s due to the timing of a new academic year getting underway, but I have been asked several times lately for advice to help new SoTL researchers get their first SoTL projects up and running. In response to these requests, I’ve constructed this list as a starting point for advice for new SoTL researchers. I’d love to see additions and expansions as comments below!

  1. Develop your research questions carefully to reflect the purpose of your inquiry. Often times, an investigator’s first foray into SoTL research is motivated by a reflection on their own teaching or by a concern regarding student learning. These are wonderful foundations for SoTL inquiry! Having a clear idea as to the purpose of your research is vitally important to creating a cohesive project. While this is not an exhaustive list, perhaps your research question might be developed to:
    • measure changes in teaching or learning over time
    • ascertain the effectiveness of a specific assessment, assignment, or pedagogical approach
    • compare groups of students across a single class or across multiple course experiences
  2. Contemplate collaboration. While you may be seeking to systematically investigate some issue specific to a course you teach or to students you work with, it can be very useful to engage collaborators for your SoTL research efforts. It might be possible to seek out peers working to answer questions similar to yours for a larger SoTL project. Or, you can recruit students to serve as research collaborators for data collection, analysis, or writing.
  3. Seek mentorship. If you have not undertaken a SoTL research study in the past, consider seeking a mentor who does have SoTL experience to help you plan and execute your study. This mentor could become a collaborator or could simply serve as an experienced voice of support and guidance in helping you though your first SoTL project. You can seek mentors though your academic unit at your university, though a teaching/learning center on your campus, or through connections with others interested in teaching and learning in your discipline.
  4. Compose a well-written IRB protocol. IRB protocols for SoTL research can be tricky to write. It is critically important that any IRB describing SoTL research adequately account for issues related to privacy, coercion, and fairness. The Office of the Cross Chair in SoTL at Illinois State University has created several resources to assist new SoTL researchers in the process of IRB design:
  5. Consider your data source(s) carefully. Your ability to collect data is limited only by your own lack of imagination! If you seek to represent student perceptions, you aren’t restricted to using a survey or questionnaire (though those can be effective!). Consider interviews or focus groups to gather data for analysis. If you’re attempting to determine the effectiveness of a pedagogical approach, consider looking at student produced artifacts or course portfolios as a potential source. If you can, seek out data from multiple sources to triangulate your findings and increase the strength of your research. Think carefully about your specific research question, determine the “best fit” data to address your question, and determine the best way to collect that data. Convenient data isn’t always good data, so be prepared to work to collect high-quality data to investigate your research question.
  6. Represent your SoTL project in a way that will be welcome in your own discipline. SoTL work can and should be represented in a way that reflects your own disciplinary practices and expectations. For some researchers, that may mean that SoTL research is shared in more “traditional” formats such as research papers, research notes, and oral/poster sessions at SoTL or disciplinary conferences. For others, SoTL can be represented though books, blog posts, wikis, and creative endeavors such as a video, a documentary, a play, a piece of artwork, or through dance. This is not meant to be an exclusive list, but simply one to make new SoTL researchers think about how they plan to share their SoTL work as that decision may influence other aspects of their research.
  7. Match your dissemination outlet(s) with your intended audience. Is the research question you’re seeking to answer with your study one that might appeal to a cross-disciplinary audience? If so, an outlet that allows a broad viewership is appropriate. Likewise, if the SoTL work you’re engaged in is most applicable within your own discipline, it would be wise to choose an outlet to share your work that is visible to your disciplinary peers. Don’t feel as though you need to only share your SoTL work in one place, however. With a bit of effort, most SoTL research can (and perhaps should!) be shared in multiple ways and forms.

If you have colleagues who are considering a SoTL research agenda, please share this list with them! And, as requested at the top of this post, please DO add any other advice you might offer to a new SoTL researcher in the comments below. We would love to expand this list to reflect the ideas of our blog’s readers!